Control method and recording medium

ABSTRACT

A method for controlling an information terminal causes a computer of the information terminal to receive, from a case retrieval system, a plurality of similar medical images having a feature quantity of a region of interest and a certain degree of similarity in accordance with the region of interest included in a target medical image, displays a display screen displaying the plurality of received similar medical images on a touch panel display, the display screen including a display region in which at least some of the plurality of received similar medical images are displayed, displays, if selection of a first similar medical image from among the at least some of the plurality of received similar medical images displayed in the display region is detected, the first similar medical image across the display region, and displays, if a swipe operation performed on the first similar medical image is detected, a second similar medical image, which has second highest similarity next to the first similar medical image among the plurality of similar medical images, in the display region such that a corresponding region of interest included in the second similar medical image is located at a certain position in the display region.

BACKGROUND 1. Technical Field

The present disclosure relates to a method for controlling aninformation terminal for retrieving medical images similar to a medicalimage to be read and a recording medium.

2. Description of the Related Art

Currently, medical imaging apparatuses for, for example, computedtomography (CT), magnetic resonance imaging (MRI), and the like arebeing developed and increasingly used. CT and MRI have made it possibleto obtain a large number of high-definition digital medical images. Inaddition, medical images read by doctors are being accumulated inpicture archiving and communication systems (PACSs) along with readingreports. For example, as disclosed in Japanese Unexamined PatentApplication Publication No. 2008-257292, a technique for retrieving pastmedical images similar to a medical image to be read from past casesaccumulated in a PACS is being developed, in order to use the pastmedical images as a reference in reading the medical image to be read.

SUMMARY

However, further improvements are necessary.

One non-limiting and exemplary embodiment achieves further improvements.

In one general aspect, the techniques disclosed here feature a methodfor controlling an information terminal that includes a touch paneldisplay and that is connected to a case retrieval system that retrievesmedical images by referring to a medical image database in which medicalimages are registered. The method includes displaying a target medicalimage, which is a medical image to be read selected from among targetmedical image candidates, on the touch panel display, causing a computerof the information terminal to detect specification informationindicating a region of interest in the target medical image, causing thecomputer of the information terminal to receive, from the case retrievalsystem, a plurality of similar medical images having a feature quantityof the region of interest and a certain degree of similarity inaccordance with the region of interest indicated by the specificationinformation, the plurality of similar medical images each including acorresponding region of interest corresponding to the region ofinterest, displaying a display screen displaying the plurality ofreceived similar medical images on the touch panel display, the displayscreen including a display region in which at least some of theplurality of received similar medical images are displayed, displaying,if selection of a first similar medical image from among the at leastsome of the plurality of received similar medical images displayed inthe display region is detected, the first similar medical image acrossthe display region, and displaying, if a swipe operation performed onthe first similar medical image is detected, a second similar medicalimage, which has second highest similarity next to the first similarmedical image among the plurality of similar medical images, in thedisplay region such that the corresponding region of interest includedin the second similar medical image is located at a certain position inthe display region.

According to the present disclosure, further improvements can beachieved.

It should be noted that general or specific embodiments may beimplemented as a system, a method, an integrated circuit, a computerprogram, a storage medium, or any selective combination thereof.

Additional benefits and advantages of the disclosed embodiments willbecome apparent from the specification and drawings. The benefits and/oradvantages may be individually obtained by the various embodiments andfeatures of the specification and drawings, which need not all beprovided in order to obtain one or more of such benefits and/oradvantages.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating an overall configuration of a hospitalinformation system adopting an information terminal according to a firstembodiment;

FIG. 2 is a block diagram illustrating configurations of the informationterminal, a medical information management system, and a case retrievalsystem;

FIG. 3 is a diagram illustrating an example of the configuration of theinformation terminal in actual implementation;

FIG. 4 is a diagram illustrating an example of a display screendisplayed on a display at a time when a doctor views a diagnosis imageusing the information terminal;

FIG. 5 is a diagram illustrating an example of a base screen displayedon the display;

FIG. 6A is a diagram illustrating an example of the base screendisplayed on the display;

FIG. 6B is a diagram illustrating an example of the base screen in adigital imaging and communications in medicine (DICOM) viewer mode;

FIG. 7 is an enlarged diagram of a disease list display region;

FIG. 8 is a diagram illustrating a base screen at a time when similarcases have been narrowed down using “mycosis”;

FIG. 9 is a diagram illustrating a base screen at a time when thesimilar cases have been narrowed down using “metastatic lung cancer”;

FIG. 10 is a diagram illustrating an example of the base screendisplayed on the display;

FIG. 11 is a diagram illustrating an example of the base screendisplayed on the display;

FIG. 12 is an enlarged diagram of a distribution list display region;

FIG. 13 is a diagram illustrating the distribution list display regionin which a checkbox for “bilateral” has been checked;

FIG. 14A is a diagram illustrating a base screen in which the similarcases have been narrowed down using a type of lesion distribution of“bilateral”;

FIG. 14B is a diagram illustrating an example of the base screendisplayed on the display;

FIG. 15 is a diagram illustrating the display list display region inwhich a checkbox for “bronchial” has been checked;

FIG. 16 is a diagram illustrating a base screen displayed when, in abase screen in which the similar cases have been narrowed down using thetype of lesion distribution of “bronchial”, a thumbnail image in a firstrow and a first column has been selected;

FIG. 17 is a diagram illustrating the distribution list display regionin which a checkbox for “subpleural” has been checked;

FIG. 18 is a diagram illustrating a base screen displayed when, in abase screen in which the similar cases have been narrowed down using thetype of lesion distribution of “subpleural”, a thumbnail image in thefirst row and the first column has been selected;

FIG. 19 is a diagram illustrating a data structure of patientinformation;

FIG. 20 is a diagram illustrating a data structure of test informationregistered in the patient information illustrated in FIG. 19;

FIG. 21 is a diagram illustrating a data structure of a medical imagedatabase;

FIG. 22 is a diagram illustrating a data structure of a diagnosisreport;

FIG. 23 is a diagram illustrating a data structure of similar case data;

FIG. 24 is a diagram schematically illustrating a region of interest setin a slice image;

FIG. 25 is a sequence diagram illustrating a process performed until thecase retrieval system receives, from the information terminal, a requestto retrieve similar cases after the information terminal requests thereading target test list from the medical information management system;

FIG. 26 is a diagram illustrating an example of a display screen for thereading target test list;

FIG. 27 is a diagram illustrating an example of the display screen forthe reading target test list after a test is selected;

FIG. 28 is a sequence diagram illustrating a process performed until theinformation terminal displays a base screen on the basis of results ofretrieval of similar cases received from the case retrieval system afterthe case retrieval system receives, from the information terminal, arequest to retrieve similar cases;

FIG. 29 is a flowchart illustrating details of a process for generatingan initial base screen indicated by S650 illustrated in FIG. 28;

FIG. 30 is a diagram illustrating a data structure of a disease listgenerated in S1300 illustrated in FIG. 29;

FIG. 31 is a diagram illustrating a first display example of the diseaselist display region;

FIG. 32 is a diagram illustrating a second display example of thedisease list display region;

FIG. 33 is a diagram illustrating a third display example of the diseaselist display region;

FIG. 34 is a diagram illustrating switching of the disease list displayregion illustrated in FIG. 32;

FIG. 35 is a diagram illustrating a data structure of a distributionlist generated in S1400 illustrated in FIG. 29;

FIG. 36 is a sequence diagram at a time when the sequence diagrams ofFIGS. 25 and 28 are seen at a level of applications;

FIG. 37 is a block diagram illustrating the information terminal, themedical information management system, and the case retrieval system ata time when a mode in which the case retrieval system extracts an imagefeature is adopted;

FIG. 38 is a sequence diagram illustrating a process performed in theconfiguration illustrated in FIG. 37 until the information terminaldisplays a base screen on the basis of results of retrieval of similarcases received from the case retrieval system after setting a region ofinterest;

FIG. 39 is a sequence diagram at a time when a process performed in theconfiguration illustrated in FIG. 37 until the information terminaldisplays a base screen on the basis of results of retrieval of similarcases received from the case retrieval system after requesting thereading target test list from the medical information management systemis seen at the level of the applications;

FIG. 40 is a sequence diagram illustrating a process for switching asimilar case displayed on the display of the information terminal on thebasis of a swipe operation;

FIG. 41 is a flowchart illustrating the process corresponding to thesequence diagram of FIG. 40;

FIG. 42 is a diagram illustrating a base screen in which a thumbnailimage displayed in a case display region is switched through a swipeoperation performed by a user;

FIG. 43 is a diagram illustrating the base screen in which the thumbnailimage displayed in the case display region is switched through the swipeoperation performed by the user;

FIG. 44 is a diagram illustrating the base screen in which the thumbnailimage displayed in the case display region is switched through the swipeoperation performed by the user;

FIG. 45 is a diagram illustrating the base screen in which the thumbnailimage displayed in the case display region is switched through the swipeoperation performed by the user;

FIG. 46 is a diagram illustrating the base screen in which the thumbnailimage displayed in the case display region is switched through the swipeoperation performed by the user;

FIG. 47 is a diagram illustrating the base screen in which the thumbnailimage displayed in the case display region is switched through the swipeoperation performed by the user;

FIG. 48 is a block diagram illustrating a detailed configuration of animage generation unit;

FIG. 49 is a diagram schematically illustrating coordinates of a displayregion determined by a display region determination section;

FIG. 50 is a sequence diagram illustrating a process performed when theuser has not performed a swipe operation for a certain period of timeafter switching a thumbnail image of a similar case by performing aswipe operation on the information terminal;

FIG. 51 is a diagram illustrating an example of a base screen displayedon the display in the sequence illustrated in FIG. 50;

FIG. 52 is a diagram illustrating an example of the base screendisplayed on the display in the sequence illustrated in FIG. 50;

FIG. 53 is a sequence diagram illustrating a process performed until athumbnail image of a next similar case is displayed after the userperforms an enlargement operation;

FIG. 54 is a flowchart illustrating the process corresponding to thesequence diagram of FIG. 53;

FIG. 55 is a diagram illustrating a pinch-out operation performed by theuser on a thumbnail image;

FIG. 56 is a diagram illustrating the pinch-out operation performed bythe user on the thumbnail image;

FIG. 57 is a diagram illustrating a base screen in which a thumbnailimage displayed in the case display region is switched through a swipeoperation performed by the user;

FIG. 58 is a diagram illustrating the base screen in which the thumbnailimage displayed in the case display region is switched through the swipeoperation performed by the user;

FIG. 59 is a diagram illustrating the base screen in which the thumbnailimage displayed in the case display region is switched through the swipeoperation performed by the user;

FIG. 60 is a block diagram illustrating a detailed configuration of animage generation unit according to a second embodiment;

FIG. 61 is a diagram schematically illustrating a relationship betweenan enlargement ratio and a display region; and

FIG. 62 is a diagram schematically illustrating a reference thumbnailimage and a thumbnail image to be displayed next before and after anenlargement process.

DETAILED DESCRIPTION Underlying Knowledge Forming Basis of the PresentDisclosure

First, a point of observation according to an aspect of the presentdisclosure will be described.

In Japanese Unexamined Patent Application Publication No. 2008-257292,an image diagnosis support apparatus is disclosed that displays a caseimage helpful in determining a disease, statistical information relatingto the disease, or the like during image diagnosis based on a diagnosistarget image. A retrieval result screen of the image diagnosis supportapparatus displays the diagnosis target image and information regardinga representative case of each disease. More specifically, the retrievalresult screen displays i) images of representative cases of top threediseases A, D, and G, ii) similarity to the diagnosis target image, thenumber of cases registered, and the number of representative cases foreach disease, iii) the number of retrieval results (the total number ofdiseases found), iv) a software button, such as a “Next Page” button,for referring to information regarding other diseases that are notdisplayed in a first page, and the like (refer to Paragraphs [0062] and[0063] and FIG. 6E of Japanese Unexamined Patent Application PublicationNo. 2008-257292).

In Japanese Unexamined Patent Application Publication No. 2008-257292,however, the retrieval result screen (refer to FIG. 6E of JapaneseUnexamined Patent Application Publication No. 2008-257292) includes a“Next Page” software button. Therefore, the “Next Page” software buttonneeds to be selected to display a next page of the retrieval resultscreen and view another representative case of each disease. If thenumber of retrieval results is large, namely, say, 200 to 300, thesoftware button needs to be operated repeatedly. In addition, theposition of a lesion does not necessarily remain the same among a largenumber of images that a doctor needs to check. Therefore, the doctorneeds to visually search for the position of a lesion each time he/shedisplays a new page. Therefore, in Japanese Unexamined PatentApplication Publication No. 2008-257292, there is a problem that adoctor needs to perform troublesome work in order to determine a diseasefor a patient using a diagnosis target image. Consequently, the doctormight not be able to identify a disease for a patient efficiently andappropriately.

In Akira Oosawa, et al. “Development of “SYNAPSE Case Match”,Content-based Image Retrieval System for Supporting Image Diagnosis”, acontent-based image retrieval system for supporting an image diagnosisgiven by a doctor is disclosed that instantaneously retrieves necessaryinformation from a clinical database constructed in a PACS or the likeand displays the information using a function of retrieving past casessimilar to an image of a lesion. More specifically, this systemretrieves case images including lesions similar to a lesion included inan inspection image and displays the cases images in order of descendingsimilarity. Thereafter, one of the displayed case images, which arereferred to as “reference case images”, is selected and displayed alongwith the inspection image (refer to p. 12 “2.2 Features of This System”and FIG. 3 of “Development of “SYNAPSE Case Match”, Content-based ImageRetrieval System for Supporting Image Diagnosis”).

In the system disclosed in “Development of “SYNAPSE Case Match”,Content-based Image Retrieval System for Supporting Image Diagnosis” ifthe displayed reference case images do not include an image to beselected, next reference case images are to be displayed. As illustratedin FIG. 3 of “Development of “SYNAPSE Case Match”, Content-based ImageRetrieval System for Supporting Image Diagnosis”, however, the size ofdisplayed reference case images is small compared to the size of theentirety of a display screen. In addition, the number of reference caseimages to be checked by the doctor is large, and the position of alesion does not necessarily remain the same in each of the large numberof case images. Therefore, the doctor needs to visually search for theposition of a lesion in each of the small reference case images eachtime he/she displays next case images. Accordingly, in “Development of“SYNAPSE Case Match”, Content-based Image Retrieval System forSupporting Image Diagnosis”, there is a problem that a doctor needs toperform troublesome work in order to determine a disease for a patientusing an inspection image. Consequently, the doctor might not be able toidentify a disease for a patient efficiently and appropriately.

When a lesion, for which a disease has not yet been identified, includedin a medical image to be read is examined, it is effective to refer toother medical images in which diseases have already been identified andthat are similar to the medical image to be read. If such a system isconstructed, however, an enormous number of medical images areregistered to a medical image database. It is desirable, even in thiscase, to effectively present similar medical images, which can be usedas a reference when a doctor gives a diagnosis using a medical image tobe read, to the doctor.

As a result of the above examination, the following aspects have beenconceived.

A first aspect of the present disclosure is a method for controlling aninformation terminal that includes a touch panel display and that isconnected to a case retrieval system that retrieves medical images byreferring to a medical image database in which medical images areregistered. The method includes displaying a target medical image, whichis a medical image to be read selected from among target medical imagecandidates, on the touch panel display, causing a computer of theinformation terminal to detect specification information indicating aregion of interest in the target medical image, causing the computer ofthe information terminal to receive, from the case retrieval system, aplurality of similar medical images having a feature quantity of theregion of interest and a certain degree of similarity in accordance withthe region of interest indicated by the specification information, theplurality of similar medical images each including a correspondingregion of interest corresponding to the region of interest, displaying adisplay screen displaying the plurality of received similar medicalimages on the touch panel display, the display screen including adisplay region in which at least some of the plurality of receivedsimilar medical images are displayed, displaying, if selection of afirst similar medical image from among the at least some of theplurality of received similar medical images displayed in the displayregion is detected, the first similar medical image across the displayregion, and displaying, if a swipe operation performed on the firstsimilar medical image is detected, a second similar medical image, whichhas second highest similarity next to the first similar medical imageamong the plurality of similar medical images, in the display regionsuch that the corresponding region of interest included in the secondsimilar medical image is located at a certain position in the displayregion.

A position of the corresponding region of interest might be different ineach of the plurality of similar medical images. On the other hand,similar medical images having the certain degree of similarity to theregion of interest of the target medical image are received. Therefore,if the position of the corresponding region of interest is different ineach of the plurality of similar medical images, a doctor needs tovisually search for the corresponding region of interest, which islocated at a different position in each of the plurality of similarmedical images, after each swipe operation. When the correspondingregions of interest of the plurality of similar medical images aresequentially observed, the doctor might need to spend unnecessary timeand effort to find the corresponding regions of interest, whichdecreases efficiency.

According to this aspect, when the second similar medical image, whichhas the second highest similarity next to the first similar medicalimage, is displayed through a swipe operation, the second similarmedical image is not simply displayed. The second similar medical imageis displayed in the display region such that the corresponding region ofinterest included in the second similar medical image is located at thecertain position in the display region.

Therefore, even when the corresponding regions of interest of the firstsimilar medical image and the second similar medical image aresequentially observed through a swipe operation, the doctor need notvisually search for the corresponding region of interest each time thedisplayed similar medical image is switched. Therefore, the doctor canconcentrate on observing the corresponding regions of interest. As aresult, when the corresponding regions of interest of the first similarmedical image and the second similar medical image are sequentiallyobserved through a swipe operation, the observation can be lessexhausting and a reading efficiency improves, thereby improving anefficiency of diagnosis.

In addition, in the first aspect, for example, if a swipe operationperformed on the second similar medical image is detected, a thirdsimilar medical image, which has third highest similarity next to thesecond similar medical image among the plurality of similar medicalimages, may be displayed in the display region such that thecorresponding region of interest included in the third similar medicalimage is located at the certain position in the display region.

According to this aspect, when the second similar medical image and thethird similar medical image are sequentially observed through a swipeoperation, the corresponding regions of interest are displayed at thecertain position. Therefore, the doctor need not visually search for thecorresponding region of interest each time the displayed similar medicalimage is switched.

In addition, in the first aspect, for example, when the first similarmedical image is displayed in the display region, the first similarmedical image may be displayed in the display region such that thecorresponding region of interest included in the first similar medicalimage is located at the certain position in the display region.

According to this aspect, the first similar medical image, which is thesimilar medical image displayed first, too, is displayed such that thecorresponding region of interest is located at the certain position.

In addition, in the first aspect, for example, the certain position inthe display region may include a position corresponding to a center ofthe display region.

According to this aspect, even if the position of the correspondingregion of interest is different between the first similar medical imageand the second similar medical image, each corresponding region ofinterest is not observed, for example, at a corner of the displayregion. Each corresponding region of interest is displayed at the centerof the display region, and regions around each corresponding region ofinterest can also be observed equally.

In addition, in the first aspect, for example, the certain position inthe display region may include a position of the corresponding region ofinterest included in the first similar medical image in the firstsimilar medical image.

According to this aspect, the position of the corresponding region ofinterest included in the first similar medical image, which is thesimilar medical image displayed first, is the certain position.Therefore, the doctor initially looks at the position of thecorresponding region of interest included in the first similar medicalimage, which is the similar medical image displayed first, and can thenobserve the corresponding region of interest included in the secondsimilar medical image without turning his/her eyes.

In addition, in the first aspect, for example, when the second similarmedical image is displayed in the display region such that thecorresponding region of interest included in the second similar medicalimage is located at the certain position in the display region, aportion of the second similar medical image outside the display regionneed not be displayed and a background image may be displayed in aportion of the display region outside the second similar medical image.

In addition, in the first aspect, for example, if a swipe operationperformed on the second similar medical image is not detected for acertain period of time after the second similar medical image isdisplayed in the display region, a display position of the secondsimilar medical image may be moved from the certain position in thedisplay region, at which the corresponding region of interest includedin the second similar medical image has been displayed, to a position atwhich entirety of the second similar medical image is displayed acrossthe display region.

If a swipe operation stops after being performed, it is likely that thedoctor is interested in a similar medical image displayed when the swipeoperation has stopped. That is, during the swipe operation, the doctoris browsing through many similar medical images, and when the swipeoperation has stopped, it is likely that the doctor is observing acurrent similar medical image.

According to this aspect, if a swipe operation is not detected for thecertain period of time, the display position of the similar medicalimage displayed when the swipe operation has stopped is moved to theposition at which the entirety of the similar medical image isdisplayed. As a result, the doctor can observe the similar medical imagedisplayed when the swipe operation has stopped in a normal displaystate. Therefore, during the swipe operation, the doctor can focus uponthe corresponding regions of interest, which improves a searchefficiency. On the other hand, when the swipe operation has stopped, thedoctor can observe all information included in a displayed similarmedical image, which improves an accuracy of diagnosis. In addition,just by stopping the swipe operation, the normal display state isestablished. Therefore, the doctor need not perform a special operationfor returning to the normal display state, which improves an efficiencyof operation.

In addition, in the first aspect, for example, the method may cause thecomputer of the information terminal to transmit information indicatingthe feature quantity of the region of interest to the case retrievalsystem and to receive, from the case retrieval system, similar medicalimages having the feature quantity of the region of interest and thecertain degree of similarity.

In addition, in the first aspect, for example, the method may cause thecomputer of the information terminal to transmit the target medicalimage and the specification information indicating the region ofinterest to the case retrieval system and to receive, from the caseretrieval system, similar medical images having the feature quantity ofthe region of interest and the certain degree of similarity obtainedfrom the target medical image and the specification information.

A second aspect of the present disclosure is a method for controlling aninformation terminal that includes a touch panel display and that isconnected to a case retrieval system that retrieves medical images byreferring to a medical image database in which medical images areregistered. The method includes displaying a target medical image, whichis a medical image to be read selected from among target medical imagecandidates, on the touch panel display, causing a computer of theinformation terminal to detect specification information indicating aregion of interest in the target medical image, causing the computer ofthe information terminal to receive, from the case retrieval system, aplurality of similar medical images having a feature quantity of theregion of interest and a certain degree of similarity in accordance withthe region of interest indicated by the specification information, theplurality of similar medical images each including a correspondingregion of interest corresponding to the region of interest, displaying adisplay screen displaying the plurality of received similar medicalimages on the touch panel display, the display screen including adisplay region in which at least some of the plurality of receivedsimilar medical images are displayed, displaying, if selection of afirst similar medical image from among the at least some of theplurality of received similar medical images displayed in the displayregion is detected, the first similar medical image across the displayregion, enlarging, if an enlargement operation performed on the firstsimilar medical image displayed across the display region is detected,the first similar medical image while moving the corresponding region ofinterest included in the first similar medical image to a positioncorresponding to a center of the display region, and enlarging, if aswipe operation performed on the enlarged first similar medical image isdetected, a second similar medical image, which has second highestsimilarity next to the first similar medical image among the pluralityof similar medical images, while moving the corresponding region ofinterest included in the second similar medical image to the center ofthe display region and matching a display size of the correspondingregion of interest included in the second similar medical image with adisplay size of the corresponding region of interest included in thefirst similar medical image and displays the second similar medicalimage in the display region.

A position of the corresponding region of interest might be different ineach of the plurality of similar medical images. On the other hand,similar medical images having the certain degree of similarity to theregion of interest of the target medical image are received. Therefore,if the position of the corresponding region of interest is different ineach of the plurality of similar medical images, a doctor needs tovisually search for the corresponding region of interest, which islocated at a different position in each of the plurality of similarmedical images, after each swipe operation. When the correspondingregions of interest of the plurality of similar medical images aresequentially observed, the doctor might need to spend unnecessary timeand effort to find the corresponding regions of interest, whichdecreases efficiency.

According to this aspect, when the second similar medical image, whichhas the second highest similarity next to the first similar medicalimage, is displayed through a swipe operation, the second similarmedical image is not simply displayed. The second similar medical imageis displayed in the display region such that the corresponding region ofinterest included in the second similar medical image is located at thecertain position in the display region.

Therefore, even when the corresponding regions of interest of the firstsimilar medical image and the second similar medical image aresequentially observed through a swipe operation, the doctor need notvisually search for the corresponding region of interest each time thedisplayed similar medical image is switched. Therefore, the doctor canconcentrate on observing the corresponding regions of interest. As aresult, when the corresponding regions of interest of the first similarmedical image and the second similar medical image are sequentiallyobserved through a swipe operation, the observation can be lessexhausting and the reading efficiency improves, thereby improving theefficiency of diagnosis.

In addition, not only the position of the correspond region of interestbut also the size of the corresponding region of interest might bedifferent in each of the plurality of similar medical images. Therefore,even if the position of each corresponding region of interest is thesame, the doctor needs to look at the second similar medical image moreclosely if the corresponding region of interest included in the secondsimilar medical image is smaller than that included in the first similarmedical image. In addition, when the corresponding regions of interestof the plurality of similar medical images are sequentially observed,the sizes of the corresponding regions of interest are different fromeach other. Therefore, the doctor might need to spend unnecessary timeand effort to find the corresponding regions of interest, whichdecreases the reading efficiency.

According to this aspect, the first similar medical image is enlargedand displayed such that the corresponding region of interest included inthe first similar medical image is located at the center of the displayregion. If a swipe operation performed on the first similar medicalimage is detected in this state, the second similar medical image, whichhas the second highest similarity next to the first similar medicalimage among the plurality of similar medical images, is enlarged whilemoving the corresponding region of interest included in the secondsimilar medical image to the center of the display region and matchingthe display size of the corresponding region of interest included in thesecond similar medical image with the display size of the correspondingregion of interest included in the first similar medical image anddisplayed in the display region.

Therefore, even when the corresponding regions of interest of the firstsimilar medical image and the second similar medical image aresequentially observed through a swipe operation, the doctor need notvisually search for the corresponding region of interest each time thedisplayed similar medical image is switched. In addition, the doctor canobserve the corresponding regions of interest of the same size. As aresult, when the corresponding regions of interest of the first similarmedical image and the second similar medical image are sequentiallyobserved through a swipe operation, the observation can be lessexhausting and the reading efficiency improves, thereby improving theefficiency of diagnosis.

In addition, in the second aspect, for example, if a swipe operationperformed on the second similar medical image is detected, a thirdsimilar medical image, which has third highest similarity among theplurality of similar medical images, may be enlarged while moving thecorresponding region of interest included in the third similar medicalimage to the center of the display region and matching a display size ofthe corresponding region of interest included in the third similarmedical image with the display size of the corresponding region ofinterest included in the first similar medical image and displayed inthe display region.

According to this aspect, when the second similar medical image and thethird similar medical image are sequentially observed through a swipeoperation, the corresponding regions of interest of the second similarmedical image and the third similar medical image can be displayed suchthat the corresponding regions of interest are located at the center ofthe display region and have the same display size.

Therefore, even when the corresponding regions of interest of the secondsimilar medical image and the third similar medical image aresequentially observed through a swipe operation, the doctor need notvisually search for the corresponding region of interest each time thedisplayed similar medical image is switched. In addition, the doctor canobserve the corresponding regions of interest of the same size. As aresult, when the corresponding regions of interest of the second similarmedical image and the third similar medical image are sequentiallyobserved through a swipe operation, the observation can be lessexhausting and the reading efficiency improves, thereby improving theefficiency of diagnosis.

In addition, in the second aspect, for example, the method may cause thecomputer of the information terminal to transmit information indicatingthe feature quantity of the region of interest to the case retrievalsystem and to receive, from the case retrieval system, similar medicalimages having the feature quantity of the region of interest and thecertain degree of similarity.

In addition, in the second aspect, for example, the method may cause thecomputer of the information terminal to transmit the target medicalimage and the specification information indicating the region ofinterest to the case retrieval system and to receive, from the caseretrieval system, similar medical images having the feature quantity ofthe region of interest and the certain degree of similarity obtainedfrom the target medical image and the specification information.

Another aspect of the present disclosure is a control method includingdisplaying a first resulting image in a first display region of a touchpanel display instead of a plurality of images including a first imageand a second image, the first resulting image being obtained byenlarging a third image included in a first region of the first image,and displaying, if the first resulting image is swiped, a secondresulting image in the first display region instead of the firstresulting image, the second resulting image being obtained by enlarginga fourth image included in a second region of the second image. Theplurality of images and a plurality of predetermined region are in aone-to-one relationship. Each of the plurality of images includes apredetermined corresponding region included in the plurality ofpredetermined regions. The first region includes a first predeterminedregion included in the plurality of predetermined regions. The thirdimage includes a fifth image included in the first predetermined region.The second region includes a second predetermined region included in theplurality of predetermined regions. The fourth image includes a sixthimage included in the second predetermined region. Among images includedin the plurality of predetermined regions, a most similar image to apredetermined image is the fifth image. Among the images included in theplurality of predetermined regions, a second most similar image to thepredetermined image is the sixth image. If the first resulting image isdisplayed in the first display region, the first predetermined region isdisplayed at a center of the first display region. If the secondresulting image is displayed in the first display region, the secondpredetermined region is displayed at the center of the first displayregion.

First Embodiment

An embodiment of the present disclosure will be described hereinafterwith reference to the drawings. In the drawings, the same components aregiven the same reference numerals.

FIG. 1 is a diagram illustrating an overall configuration of a hospitalinformation system adopting an information terminal according to thisembodiment. As illustrated in FIG. 1, the hospital information systemincludes an information terminal 100, a medical information managementsystem 200, and a case retrieval system 300.

The information terminal 100, the medical information management system200, and the case retrieval system 300 are communicably connected to oneanother through a network 400.

The medical information management system 200 and the case retrievalsystem 300 need not necessarily be installed in a hospital, but may besoftware operating on a data center, a private cloud server, a publiccloud server, or the like outside the hospital. If the medicalinformation management system 200 and the case retrieval system 300 areinstalled in the hospital, the network 400 may be a local area network(LAN). The LAN may be a wired LAN of an Institute of Electrical andElectronics Engineers (IEEE) 802.3 series, a wireless LAN of an IEEE802.11 series, or a network including both types of LAN. If the medicalinformation management system 200 and the case retrieval system 300 arerealized using a server outside the hospital, the network 400 may be theInternet.

The information terminal 100 may be an information terminal such as apersonal computer (PC) or a tablet terminal. The medical informationmanagement system 200 may be a PACS, an electronic medical recordsystem, or the like.

FIG. 2 is a block diagram illustrating configurations of the informationterminal 100, the medical information management system 200, and thecase retrieval system 300. As illustrated in FIG. 2, the informationterminal 100 includes a display 101, an operation unit 102, an inputcontrol unit 103, a display control unit 104, a region of interest (ROI)management unit 105, a disease list management unit 108, a distributionlist management unit 109, a communication control unit 110, and an imagegeneration unit 112.

The display 101 is, for example, a liquid crystal monitor including atouch panel and displays a medical image used for diagnosis and amedical record image along with a report input screen for inputting adiagnosis and the like.

The operation unit 102 is, for example, a touch panel and receivesvarious operations performed by a user on the information terminal 100.For example, the operation unit 102 receives an operation performed bythe user on a medical image or a medical record image displayed on thedisplay 101, an operation for inputting a diagnosis in the report inputscreen, and the like.

The input control unit 103 detects an operation performed by the user onthe operation unit 102. The input control unit 103 then interprets theoperation and transmits a result of the operation to another component.For example, the input control unit 103 receives, from the operationunit 102, data regarding coordinates of a position at which an object(for example, the user's finger) has touched the touch panel. The inputcontrol unit 103 then compares the data with coordinates of graphicaluser interfaces (GUIs) generated by the display control unit 104, inorder to determine whether, among the GUIs, a button object has beenselected. The input control unit 103 transmits a result of the operationperformed by the user to another component.

The display control unit 104 generates GUIs of the information terminal100 and displays the GUIs on the display 101.

The ROI management unit 105 generates, before retrieval of similarcases, ROI information indicating a ROI set in a retrieval query image,which will be described later, and stores the ROI information in amemory, in order to manage the ROI information.

The disease list management unit 108 generates a disease list (FIG. 30)of similar cases displayed in a case display region 710 (FIG. 5) andstores the disease list in the memory, in order to manage the diseaselist.

The distribution list management unit 109 generates a distribution list(FIG. 35) indicating the lesion distribution of the similar casesdisplayed in the case display region 710 and stores the distributionlist in the memory, in order to manage the distribution list.

The communication control unit 110 includes, for example, acommunication device for connecting the information terminal 100 to thenetwork 400 and controls communication between the information terminal100 and the medical information management system 200 and between theinformation terminal 100 and the case retrieval system 300. In addition,the communication control unit 110 receives requests to transmit variouspieces of data from other blocks and transmits the various pieces ofdata to the medical information management system 200 or the caseretrieval system 300. The communication control unit 110 also receivesdata transmitted from the medical information management system 200 orthe case retrieval system 300 and transfers the data to appropriateblocks.

The image generation unit 112 obtains thumbnail images of similar casescreated in advance. The image generation unit 112 generates thumbnailimages whose display regions on the display 101 have been changed inaccordance with an operation performed by the user and outputs thegenerated thumbnail images to the display control unit 104.

The image generation unit 112 obtains, from the input control unit 103,a result of an operation performed by the user on the operation unit102. The image generation unit 112 receives similar case data(similarity and ROI information) from a similar case retrieval unit 303through the communication control unit 110. If the user performs a swipeoperation, the image generation unit 112 generates a thumbnail image ofa similar case such that center coordinates of a ROI of the thumbnailimage is located at a center position of a display regions in which thethumbnail image is displayed. The generation of a thumbnail image basedon a swipe operation will be described in detail later.

As illustrated in FIG. 2, the medical information management system 200includes a patient information accumulation unit 201, a patientinformation management unit 202, a medical image data accumulation unit203, a medical image data management unit 204, a diagnosis reportmanagement unit 205, and a communication control unit 206.

The patient information accumulation unit 201 accumulates patientinformation 1000 (FIG. 19) in which personal information such as thegender and age of each patient, clinical information such as pastmedical histories, and test information regarding a blood test or thelike are registered.

The patient information management unit 202 performs a process forupdating the patient information 1000 (FIG. 19) accumulated in thepatient information accumulation unit 201 by registering data input fromthe user, a process for outputting the patient information 1000 to thedisplay control unit 104, and the like, in order to manage the patientinformation 1000. The medical image data accumulation unit 203accumulates medical image data, which includes test images of patients.

The medical image data management unit 204 accumulates the medical imagedata in the medical image data accumulation unit 203 in order to managethe medical image data.

The diagnosis report management unit 205 manages a diagnosis report 3000(FIG. 22) indicating diagnoses given by doctors as a result of variousmedical tests conducted on patients.

The communication control unit 206 includes, for example, acommunication device for connecting the medical information managementsystem 200 to the network 400. The communication control unit 206receives requests to transmit various pieces of data from other blocksand transmits the various pieces of data to the information terminal 100or the case retrieval system 300. The communication control unit 206also receives data transmitted from the information terminal 100 or thecase retrieval system 300 and transfers the data to appropriate blocks.

As illustrated in FIG. 2, the case retrieval system 300 includes asimilar case data accumulation unit 301, an image feature extractionunit 302, and a similar case retrieval unit 303.

The similar case data accumulation unit 301 accumulates similar casedata 4000 (FIG. 23) in which, among the similar cases managed by themedical information management system 200, image features extracted froma large number of similar cases selected as data to be retrieved assimilar cases, generated thumbnail images, and the like are registered.

The image feature extraction unit 302 extracts an image feature includedin ROI information regarding a retrieval query image transmitted fromthe communication control unit 110 of the information terminal 100.

The ROI information is an example of specification informationindicating a ROI.

The similar case retrieval unit 303 generates results of retrieval ofsimilar cases by comparing the image feature extracted by the imagefeature extraction unit 302 with image features of one or more similarcases accumulated in the similar case data accumulation unit 301.

A communication control unit 304 includes, for example, a communicationdevice for connecting the case retrieval system 300 to the network 400.The communication control unit 304 receives requests to transmit variouspieces of data from other blocks and transmits the various pieces ofdata to the information terminal 100 or the medical informationmanagement system 200. The communication control unit 304 also receivesdata transmitted from the information terminal 100 or the medicalinformation management system 200 and transfers the data to appropriateblocks.

FIG. 3 is a diagram illustrating an example of the configuration of theinformation terminal 100 in actual implementation. As illustrated inFIG. 3, the information terminal 100 includes applications 501, anoperating system (OS) 502, a memory 503, and other pieces of hardwarethat are not illustrated.

The applications 501 are application software for causing a PC or atablet terminal to function as the information terminal 100 and executedby a processor of the information terminal 100. The applications 501 maybe installed in the information terminal 100 by reading the applications501 from a computer-readable recording medium or downloading theapplications 501 from the network 400.

Here, the applications 501 include a medical information managementapplication and a similar case retrieval application. The medicalinformation management application is an application for enabling theinformation terminal 100 to cooperate with the medical informationmanagement system 200. The similar case retrieval application is anapplication for enabling the information terminal 100 to cooperate withthe case retrieval system 300. Both applications communicate data witheach other in order to integrate services provided by the medicalinformation management system 200 and services provided by the caseretrieval system 300 with each other in the information terminal 100.

The OS 502 is basic software of the information terminal 100 andexecuted by the processor of the information terminal 100. The memory503 may be a storage device such as random-access memory (RAM) or aread-only memory (ROM) included in the information terminal 100 andstores datasets included in the applications 501.

By executing the applications 501 using the processor of the informationterminal 100, the functions of the input control unit 103, the displaycontrol unit 104, the ROI management unit 105, the disease listmanagement unit 108, the distribution list management unit 109, thecommunication control unit 110, and the image generation unit 112 arerealized.

In this embodiment, however, only the applications 501 may be installedin the information terminal 100, or the applications 501 and the OS 502may be installed. Alternatively, the applications 501, the OS 502, andthe memory 503 may be installed in the information terminal 100, or theapplications 501, the OS 502, the memory 503, and the other pieces ofhardware that are not illustrated may be installed. In any case, theinformation terminal 100 according to this embodiment can be realized.

FIG. 4 is a diagram illustrating an example of a display screen K1displayed on the display 101 when a doctor views a diagnosis image usingthe information terminal 100. As illustrated in FIG. 4, a base screen ofthe information terminal 100 displayed on the display 101 includes amedical image viewer 705, a reading target test list button 701, a ROIinput button 702, a similar case retrieval button 703, a reading reportinput button 704, and a scroll bar 706.

Medical images are normally recorded in a DICOM format. The medicalimage viewer 705 is a viewer capable of handling the DICOM format. It isassumed in this embodiment that medical images are chest CT images,which are a large number of tomograms (hereinafter also referred to as“slice images”). However, this is an example, and CT images of anotherpart of a body (for example, head, abdomen, leg, or arm) may be adopted.

If the user performs an operation on the touch panel, another sliceimage is displayed on the medical image viewer 705 as a chest CT image.Here, among the chest CT images arranged on the touch panel, forexample, first slice images indicate the neck, and last slice imagesindicate the abdomen.

As medical images, MRI images or plain roentgenograms may be adoptedinstead of chest CT images. Although the number of medical imagesdisplayed on the medical image viewer 705 is 1 in the exampleillustrated in FIG. 4, this is just an example. Another number ofmedical images, such as 2 or 4, may be displayed. If the number ofmedical images displayed on the medical image viewer 705 is large, thenumber of medical images that can be compared with one another at onceis large, but the area of each image becomes small. Therefore, thenumber of medical images displayed may vary in accordance with thedisplay size of the display 101. The user or a manager may arbitrarilychange the number of medical images displayed.

Before the similar case retrieval application is activated, the medicalimage viewer 705 displays a chest CT image of a certain patient used foractivating retrieval of similar cases. In particular, an image (that is,a retrieval query image) at a slice position when a reader activates theretrieval of similar cases is displayed as a chest CT image. The displaycontrol unit 104 may superimpose, upon the retrieval query image, a ROIwith which the retrieval of similar cases is performed. The retrievalquery image is an example of a target medical image, which is a medicalimage to be read.

The reading target test list button 701 is a button for switching thedisplay screen of the display 101 from the display screen K1 for adiagnosis image illustrated in FIG. 4 to a display screen K5(illustrated in FIG. 26, which will be referred to later) for a readingtarget test list from which a reading target is selected.

The ROI input button 702 is a button for, before the doctor retrievessimilar cases, establishing a state in which a ROI can be input to theretrieval query image displayed on the medical image viewer 705. Byinputting coordinates on the touch panel after tapping the ROI inputbutton 702, the doctor can input the ROI illustrated in FIG. 4.

The similar case retrieval button 703 is a button for activating thesimilar case retrieval application. By tapping the similar caseretrieval button 703 after inputting a ROI, the user can activate thesimilar case retrieval application.

The reading report input button 704 is a button for switching thedisplay screen from the display screen K1 for a diagnosis imageillustrated in FIG. 4 to a creation screen for creating the diagnosisreport 3000 (illustrated in FIG. 22, which will be referred to later).

The reading target test list button 701, the ROI input button 702, thesimilar case retrieval button 703, and the reading report input button704 are examples. The same operations as described above may beperformed using any input method other than that in which the buttonsare used, instead.

The scroll bar 706 is used for switching the medical image displayed onthe medical image viewer 705 to another slice image. The display controlunit 104 displays a slice image located at a position corresponding to aposition of the scroll bar 706.

For example, if the user touches the scroll bar 706 with an object (forexample, the user's finger) and moves the object in contact with thescroll bar 706 upward or downward, the input control unit 103 detectsthe touch and the movement. The display control unit 104 moves, inaccordance with the movement of the object, a displayed position of thescroll bar 706. In addition, the display control unit 104 switches theslice image displayed on the medical image viewer 705 to a slice imagelocated at a position corresponding to the position of the scroll bar706.

“24/82” is displayed in a slice position display portion 705Aillustrated in FIG. 4. This means that a medical image displayed on themedical image viewer 705 illustrated in FIG. 4 is a 24th slice imageamong a total of 82 slice images.

FIG. 5 is a diagram illustrating an example of a base screen K2displayed on the display 101 after the information terminal 100activates the similar case retrieval application. As illustrated in FIG.5, after the similar case retrieval application is activated, thedisplay screen K1 (FIG. 4) for a medical image during diagnosis (atarget medical image, which is a medical image to be read) switches tothe base screen K2 displaying thumbnail images of similar cases. Asillustrated in FIG. 5, the base screen K2 (an example of a displayscreen) displayed on the display 101 includes a target display region708, a case display region 710 (an example of a display region), adisease list display region 730, and a distribution list display region750.

The target display region 708 is a region in which a retrieval queryimage (target medical image) used for retrieving similar cases isdisplayed.

The case display region 710 is a region in which thumbnail images ofcases similar to the retrieval query image are displayed in order ofdescending similarity. Here, the thumbnail images of the similar casesare an example of similar medical images.

For example, it is likely that the display 101 including the touchpanel, such as a tablet, does not have a large display region. In thiscase, if thumbnail images of a large number of similar cases aredisplayed, the size of each thumbnail image becomes small, therebymaking it difficult to view the similar cases. Therefore, in a basescreen K3, the case display region 710 includes only one thumbnail imageof a similar case that is most similar to the retrieval query image asillustrated in FIG. 6A (referred to later).

The case display region 710 displays images obtained by converting theresolution and pixel values of DICOM images. Since a large number ofsimilar cases are displayed in the case display region 710, it takestime to complete processing if the conversion of resolution and pixelvalues is performed each time the similar cases are displayed.Therefore, thumbnail images in an initial screen are created fromoriginal DICOM images in advance and saved in the case retrieval system300.

The conversion of resolution and pixel values will be briefly describedhereinafter. Whereas the resolution of an original CT image (sliceimage) is 512×512 pixels, the resolution of a thumbnail image is lower.Therefore, resolution needs to be converted. A thumbnail image isgenerated by performing a low-resolution process and a tone conversionprocess on an original CT image (slice image).

The tone conversion process is performed as follows. In a DICOM image(slice image) obtained as a result of CT, each pixel value (CT value) isa value indicating one of 2,000 tones, namely −1,000 to +1,000 (inHounsfield units (HUs)). Therefore, it is difficult to display the DICOMimage on a common 8-bit grayscale display. Even if the DICOM image canbe displayed, it is difficult for a person to visually distinguish anemphysema region (CT value: −1,000 HU), ordinary tissues in a lung field(CT value: about −900 HU), a frosted glass region (CT value: −800 HU),soft tissues (CT value: −100 to −50 HU), water (CT value: 0 HU), andbone (CT value: 1,000 HU) from one another in 2,000 tones.

Therefore, in a slice image, a window level and a window width areusually set for each pixel value in order to reconfigure and displayeach pixel value as an 8-bit pixel value. Here, the window level refersto a CT value as the center of a window, and the window width refers toa width from the center of the window.

For example, if a DICOM image is reconfigured under a lung fieldcondition, the window level is set to −550 to −800 and the window widthis set to 1,000 to 1,600. Therefore, the pixel values of an originalslice image are reduced to the 8-bit pixel values of a resultantthumbnail image through the above-described process.

The thumbnail images displayed in the case display region 710 of thebase screen K2 (FIG. 5) are thumbnail images indicating similar caseswhose distances to a feature vector of a diagnosis target case areshorter than or equal to a certain threshold. In the base screen K3(FIG. 6A, which will be referred to later), the thumbnail imagedisplayed in the case display region 710 is a thumbnail image indicatinga similar case whose a distance to the feature vector of the diagnosistarget case is the smallest. Here, for example, Euclidean distance isused as distance. Alternatively, another distance scale such as cityblock distance may be adopted as distance. The shorter the distance is,the more similar the two images are to each other. As the featurevector, not one obtained from a thumbnail image but one obtained from anoriginal DICOM image is used.

In FIG. 5, for example, in an upper-right portion of the base screen K2,a retrieval result number display region 714 is provided. The retrievalresult number display region 714 displays the number of cases similar toa diagnosis target case obtained from the case retrieval system 300 as aresult of a retrieval process.

If the number of similar cases is large, it is difficult for the casedisplay region 710 to display all the similar cases at once. Therefore,a vertically long scroll bar 716 is provided, for example, to the rightof the case display region 710. The display control unit 104 displaysupper or lower thumbnail images included in the case display region 710in accordance with the amount of movement of the scroll bar 716. As aresult, the user can display similar cases that have not been displayedin the case display region 710 and observe such similar cases.

Alternatively, the scroll bar 716 may be horizontally long. In thiscase, the display control unit 104 displays left or right thumbnailimages displayed in the case display region 710 in accordance with theamount of movement of the scroll bar 716.

Although it is assumed that the information terminal 100 obtainsthumbnail images whose distances to a retrieval query image are shorterthan or equal to a certain threshold from the case retrieval system 300,this is just an example. For example, the information terminal 100 mayalways obtain a certain number of thumbnail images from the caseretrieval system 300 in order of descending similarity, instead.Alternatively, the information terminal 100 may always obtain thumbnailimages from the case retrieval system 300 such that a certain number ofthumbnail images of a certain confirmed disease are included.

As a method for displaying thumbnail images in the case display region710, for example, a display method may be adopted in which thumbnailimages are displayed from left to right in order of ascending distance,with a thumbnail image whose a distance to a retrieval query image isthe shortest displayed at a leftmost position in a first row, and whenthe first row no longer has space for another thumbnail image, a nextthumbnail image is displayed at a leftmost position in a second row.That is, in the case display region 710, a display method may be adoptedin which thumbnail images are displayed in order of ascending distancefrom an upper left portion to a lower right portion.

Another display method may obviously be adopted in this embodiment. Forexample, a display method may be adopted in which thumbnail images aredisplayed from top to bottom in order of ascending distance, with athumbnail image whose distance to a retrieval query image is theshortest displayed at an uppermost position in a first column, and whenthe first column no longer has space for another thumbnail image, a nextthumbnail image is displayed at an uppermost position in a secondcolumn. Alternatively, a configuration in which the user can switchbetween these display methods may be adopted.

Although a distance to a feature vector is adopted as distance in theabove example, a distance to a feature quantity may be adopted, instead.Although distance is adopted as similarity in the above example, anymeasure may be adopted insofar as the measure indicates similaritybetween images, such as cosine similarity. If cosine similarity isadopted, two images are more similar to each other as a value becomescloser to 1.

The similar cases displayed in the case display region 710 may benarrowed down on the basis of a disease displayed in the disease listdisplay region 730 or a type of lesion distribution displayed in thedistribution list display region 750, details of which will be describedlater. Current retrieval conditions of similar cases are displayed in adisplay condition display region 715. The example illustrated in FIG. 5illustrates a state immediately after retrieval of similar cases, thatis, thumbnail images have not been narrowed down. Therefore, the displaycondition display region 715 displays “all diseases”.

FIG. 6A is a diagram illustrating an example of the base screen K3displayed on the display 101. For example, if the user selects athumbnail image by double-tapping the thumbnail image with an object(for example, the user's finger) in the base screen K2 illustrated inFIG. 5, the input control unit 103 detects the selection. The displaycontrol unit 104 displays, as illustrated in FIG. 6A, the base screen K3in which the selected thumbnail image is displayed across the casedisplay region 710 on the display 101.

In a lower part of the case display region 710 illustrated in FIG. 6A, aconfirmed diagnosis display region 711, a distance display region 712,and a similarity ranking display region 713 are provided. In theconfirmed diagnosis display region 711, a confirmed diagnosis of atarget similar case is displayed. The confirmed diagnosis refers to adisease confirmed through a diagnosis in a target similar case. In thedistance display region 712, a distance between a feature vector of aslice image in the target similar case and a feature vector of aretrieval query image is displayed. In the similarity ranking displayregion 713, a ranking of the target similar case in terms of similarityto the retrieval query image is displayed.

In the example illustrated in FIG. 6A, “lung abscess” displayed in theconfirmed diagnosis display region 711 means that the thumbnail image isa thumbnail image indicating a similar case in which a diagnosis of lungabscess is confirmed. “0.05” displayed in the distance display region712 means that the distance between the slice image of the similar caseand the retrieval query image is 0.05. “1/62” displayed in thesimilarity ranking display region 713 means that the thumbnail image ofthe similar case is a most similar image to the retrieval query imageamong thumbnail images of 62 similar cases. That is, the exampleillustrated in FIG. 6A indicates that the user has selected a thumbnailimage in a first row and a first column in the base screen K2illustrated in FIG. 5.

As illustrated in FIG. 6A, the thumbnail image displayed in the casedisplay region 710 includes a corresponding ROI (CROI). The CROI is aregion (that is, a region similar to the ROI) corresponding to the ROI(FIG. 4) set in the retrieval query image (the medical image to beread). In the following description, the CROI will also be simplyreferred to as a “ROI”.

In an upper part of the case display region 710 illustrated in FIG. 6A,a scroll bar 707 is provided. By performing a swipe operation, thesimilar case displayed in the case display region 710 can be changed toanother similar case. The switching of a similar case through a swipeoperation will be described in detail later.

The scroll bar 707 indicates a position of the similarity of the similarcase displayed in the case display region 710 among all the retrievedsimilar cases. In the example illustrated in FIG. 6A, as indicated by“1/62” displayed in the similarity ranking display region 713, thesimilar case having the highest similarity is displayed in the casedisplay region 710. Therefore, the scroll bar 707 is located at aleftmost position thereof.

In the base screen K3 illustrated in FIG. 6A, for example, if the userselects the thumbnail image displayed in the case display region 710 bytapping the thumbnail image with an object, the input control unit 103detects the selection. The display control unit 104 displays the basescreen K2 illustrated in FIG. 2 on the display 101.

FIG. 6B is a diagram illustrating an example of a base screen K4 in aDICOM viewer mode. In the base screen K3 illustrated in FIG. 6A, forexample, if the user selects the thumbnail image displayed in the casedisplay region 710 by double-tapping the thumbnail image with an object,the input control unit 103 detects the selection. The display controlunit 104 displays, as illustrated in FIG. 6B, the base screen K4 in theDICOM viewer mode on the display 101.

As illustrated in FIG. 6B, in an upper part of a thumbnail imagedisplayed in the case display region 710, a slice number display region710A and a mode display region 710B are provided. In addition, thescroll bar 706 is provided to the right of the case display region 710.“Mode: DICOM Viewer” displayed in the mode display region 710B meansthat a current mode is the DICOM viewer mode.

In the slice number display region 710A, a slice number of the currentlydisplayed thumbnail image is displayed. In the example illustrated inFIG. 6B, “32/75” is displayed in the slice number display region 710A.The slice number display region 710A illustrated in FIG. 6B indicatesthat the similar case illustrated in FIG. 6B (that is, the similar casein the first row and the first column illustrated in FIG. 5) includes 75slice images. The slice number display region 710A illustrated in FIG.6B also indicates that the thumbnail image illustrated in FIG. 6B is athirty-second slice image among the 75 slice images.

Now, a purpose of use of the DICOM viewer mode illustrated in FIG. 6Bwill be described. As illustrated in FIG. 5, in the base image K2displayed after the similar case retrieval application is activated, theslice images, each including a CROI, are displayed as thumbnail images.If the user desires to check previous and next slice images of a sliceimage including the CROI, the user displays the base screen K4 in theDICOM viewer mode illustrated in FIG. 6B.

In FIG. 6B, if the user touches the scroll bar 706 with an object andmoves the object in contact with the scroll bar 706 upward or downward,the input control unit 103 detects the touch and the movement. Thedisplay control unit 104 moves the display position of the scroll bar706 in accordance with the movement of the object. In addition, thedisplay control unit 104 switches the thumbnail image displayed in thecase display region 710 to a slice image corresponding to the positionof the scroll bar 706. As a result of this operation, the user can checkthe previous and next slice images of the slice image including theCROI.

In the base screen K4 in the DICOM viewer mode illustrated in FIG. 6B,for example, if the user selects the thumbnail image displayed in thecase display region 710 by double-tapping the thumbnail image with anobject, the input control unit 103 detects the selection. The displaycontrol unit 104 displays the base screen K3 illustrated in FIG. 6A onthe display 101. Here, for example, a “Return” button may be displayedin the base screen K4 in the DICOM viewer mode illustrated in FIG. 6B.If the “Return” button is selected, the base screen K4 in the DICOMviewer mode illustrated in FIG. 6B may return to the base screen K3illustrated in FIG. 6A.

In FIG. 5, in an upper left portion of the base screen K2, the diseaselist display region 730 having a title “disease list” is provided. Inthe disease list display region 730, confirmed diagnoses of all similarcases, which have been obtained as a result of the retrieval of similarcases, are displayed. After a diagnosis is made and a disease isidentified through the diagnosis, a diagnosis target case is accumulatedin the case retrieval system 300 as a similar case. Therefore, eachsimilar case includes a confirmed diagnosis.

FIG. 7 is an enlarged diagram of the disease list display region 730. InFIG. 7, confirmed diagnoses are classified on the basis of largecategories (731, 734, 737, 741, and 744) and small categories (732, 733,735, 736, 738, 739, 740, 742, 743, and 745). In the example illustratedin FIG. 7, mycosis 731, neoplastic 734, nonneoplastic 737,mycobacteriosis 741, and others 744 are displayed as the largecategories.

In the example illustrated in FIG. 7, aspergillosis 732 andcryptococcosis 733 are displayed as small categories of the mycosis 731.Lung cancer 735 and metastatic lung cancer 736 are displayed as smallcategories of the neoplastic 734. Lung abscess 738, sarcoidosis 739, andseptic emboli 740 are displayed as small categories of the neoplastic737. Nontuberculous mycobacteria (NTM) 742 and tuberculosis 743 aredisplayed as small categories of the mycobacteriosis 741. Bronchiectasis745 is displayed as a small category of the others 744.

In addition, the number of cases of each category is displayed next toeach of the large categories and the small categories. By selecting oneof the large categories and the small categories displayed in thedisease list display region 730, the user can narrow down the similarcases displayed in the case display region 710. As illustrated in FIG.5, the 62 similar cases including various diseases are displayedimmediately after the retrieval of similar cases, but if the user tapsthe mycosis 731 illustrated in FIG. 7 with an object, the displaycontrol unit 104 displays only similar cases of mycosis in the casedisplay region 710 as illustrated in FIG. 8. If the user taps themetastatic lung cancer 736 illustrated in FIG. 7 with an object, thedisplay control unit 104 displays only similar cases of metastatic lungcancer in the case display region 710 as illustrated in FIG. 9.

At this time, the display control unit 104 displays a category in thedisplay condition display region 715 so that the user can understandwhich category the similar cases currently displayed in the case displayregion 710 belong to. FIG. 8 is a diagram illustrating the base screenK2 at a time when the similar cases have been narrowed down using“mycosis”. FIG. 9 is a diagram illustrating the base screen K2 at a timewhen the similar cases have been narrowed down using “metastatic lungcancer”.

In the example illustrated in FIG. 8, since “mycosis” has been used fornarrowing down the similar cases, “mycosis” is displayed in the displaycondition display region 715. In the example illustrated in FIG. 9,since “metastatic lung cancer” has been used for narrowing down thesimilar cases, “metastatic lung cancer” is displayed in the displaycondition display region 715.

At this time, the display control unit 104 displays the number ofsimilar cases currently displayed in the case display region 710 in theretrieval result number display region 714 so that the user canunderstand the number of similar cases. In the example illustrated inFIG. 8, because the number of similar cases falling into the category of“mycosis” is 14, “14 results” is displayed in the retrieval resultnumber display region 714. In the example illustrated in FIG. 9, becausethe number of similar cases falling into the category of “metastaticlung cancer” is 3, “3 results” is displayed in the retrieval resultnumber display region 714.

Because of this function, only similar cases falling into a categorydetermined by the doctor as a target of image diagnosis are displayed inthe case display region 710. Therefore, the doctor can easily checkwhether a diagnosis target case contradicts a disease presumed thereby.

FIGS. 10 and 11 are diagrams illustrating examples of the base screen K3displayed on the display 101. In the base screen K2 illustrated in FIG.8, for example, if the user selects the thumbnail image in the first rowand the first column by double-tapping the thumbnail image with anobject, the input control unit 103 detects the selection. The displaycontrol unit 104 displays, as illustrated in FIG. 10, the base screen K3in which the selected thumbnail image is displayed across the casedisplay region 710 on the display 101. Similarly, in the base screen K2illustrated in FIG. 9, for example, if the user selects a thumbnailimage in a first row and a first column, the base screen K3 illustratedin FIG. 11 is displayed.

The base screen K3 illustrated in FIG. 10 or 11 is a screen similar tothe base screen K3 illustrated in FIG. 6A. Switching of a screen isperformed in the same manner as that described above.

That is, in the base screen K3 illustrated in FIG. 10, for example, ifthe user selects the thumbnail image displayed in the case displayregion 710 by tapping the thumbnail image with an object, the inputcontrol unit 103 detects the selection. The display control unit 104displays the base screen K2 illustrated in FIG. 8 on the display 101.Similarly, if the user taps the thumbnail image in the base screen K3illustrated in FIG. 11, the base screen K2 illustrated in FIG. 9 isdisplayed.

In the base screen K3 illustrated in FIG. 10 or 11, if the user selectsthe thumbnail image displayed in the case display region 710 bydouble-tapping the thumbnail image with an object, the input controlunit 103 detects the selection. The display control unit 104 displaysthe base screen K4 in the DICOM viewer mode on the display 101.

In FIG. 5, in the middle of a left part of the base screen K2, thedistribution list display region 750 having a title “lesiondistribution” is provided. In the distribution list display region 750,types of lesion distribution of all the similar cases obtained from thecase retrieval system 300 as a result of the retrieval of similar casesare displayed.

FIG. 12 is an enlarged diagram of the distribution list display region750. In the example illustrated in FIG. 12, names of seven types oflesion distribution are displayed, and a checkbox is provided to theleft of the name of each type of region distribution. In the exampleillustrated in FIG. 12, diffuse 751, segmental 752, bronchial 753,bilateral 754, multiple 755, subpleural 756, and hematogenous 757 aredisplayed as the types of lesion distribution.

These types of lesion distribution are defined in advance, anddistribution flag values (1 for applicable and 0 for inapplicable)indicating whether these types of lesion distribution, namely thediffuse 751 to the hematogenous 757, apply are given to each similarcase. All the distribution flag values might be 0 for some similarcases, whereas a plurality of distribution flag values might be 1 forother similar cases.

The case retrieval system 300 according to this embodiment retrievessimilar cases including ROIs similar to a ROI set by the user in a sliceimage of a diagnosis target case. A lesion might exist in slice imagesother than the slice image in which the user has set a ROI. Afterretrieving similar cases using the slice image in which the user has seta ROI, the user might desire to compare slice images other than theslice image in which the user has set a ROI with the retrieved similarcases. In this case, the user displays other slice images by performingan operation for displaying the other slice images in the medical imageviewer 705 (FIG. 4), in order to compare the other slice images with theretrieved similar cases. If, among all the retrieved similar cases, onlysimilar cases relating to a target lesion are displayed in the casedisplay region 710, slice images including the desired lesion can besmoothly extracted from the slice images other than the slice image inwhich the user has set a ROI. Therefore, in this embodiment, a functionof narrowing down the retrieved similar cases on the basis of a desiredtype of lesion distribution is provided in order to smoothly performthis operation.

In this embodiment, as types of lesion distribution in the lung field,the diffuse 751 to the hematogenous 757 illustrated in FIG. 12 areadopted. As illustrated in FIG. 12, the display control unit 104displays the checkboxes and the names of the types of lesiondistribution such that types of lesion distribution with which thesimilar cases can be narrowed down are displayed in an active state anda type of lesion distribution with which the similar cases cannot benarrowed down is displayed in an inactive state. Here, luminanceindicating the active state is higher than luminance indicating theinactive state. The luminance indicating the inactive state is lowerthan the luminance indicating the active state.

In the example illustrated in FIG. 12, the diffuse 751 and the bronchial753 to the hematogenous 757 are displayed in the active state, and thesegmental 752 is displayed in the inactive state. This is because thedistribution flag values of the diffuse 751 and the bronchial 753 to thehematogenous 757 are 1 (applicable) in at least one of all the similarcases obtained as a result of the retrieval of similar cases, and thedistribution flag value of the segmental 752 is 0 (inapplicable) in allthe obtained similar cases.

If the input control unit 103 detects that one or more checkboxes in theactive state are checked, the display control unit 104 displays, in thecase display region 710, only similar cases corresponding to the typesof lesion distribution whose checkboxes have been checked.

With respect to the segmental 752, the distribution flag value is 0(inapplicable) in all the similar cases obtained as a result of theretrieval. Therefore, if the checkbox for the segmental 752 can bechecked, it is of no use to check the checkbox because no similar casesare displayed in the case display region 710. Therefore, in thisembodiment, types of lesion distribution whose distribution flag valuesare 0 (inapplicable) in all the similar cases obtained as a result ofthe retrieval are displayed in the inactive state in order to avoid sucha situation.

FIG. 13 is a diagram illustrating the distribution list display region750 in which the checkbox for the bilateral 754 has been checked. FIG.14A is a diagram illustrating the base screen K2 in which the similarcases have been narrowed down using the type of lesion distribution ofthe bilateral 754. As illustrated in FIG. 13, if the checkbox for thebilateral 754 is checked, the display control unit 104 displays, asillustrated in FIG. 14A, only similar cases including the type of lesiondistribution of bilateral in the case display region 710. In this case,the number of similar cases including the type of lesion distribution ofbilateral is 10. Therefore, the display control unit 104 displays “10results” in the retrieval result number display region 714. The displaycontrol unit 104 also displays the name of a displayed disease and thename of the type of lesion distribution, “bilateral”, in the displaycondition display region 715. Because the similar cases have not beennarrowed down on the basis of a category displayed in the disease listdisplay region 730 in the example illustrated in FIG. 14A, “alldiseases” is displayed in the display condition display region 715.

FIG. 14B is a diagram illustrating an example of the base screen K3displayed on the display 101. In the base screen K2 illustrated in FIG.14A, for example, if the user selects a thumbnail image in a first rowand a first column by double-tapping the thumbnail image with an object,the input control unit 103 detects the selection. The display controlunit 104 displays, as illustrated in FIG. 14B, the base screen K3 inwhich the selected thumbnail image is displayed across the case displayregion 710 on the display 101. The base screen K3 illustrated in FIG.14B is similar to the base screen K3 illustrated in FIG. 6A. Switchingof a screen is performed in the same manner as that described above.

Similarly, if the checkbox for the multiple 755 is checked, the displaycontrol unit 104 displays only similar cases including the type oflesion distribution of multiple in the case display region 710.Similarly, if the checkbox for the diffuse 751 is checked, the displaycontrol unit 104 displays only similar cases including the type oflesion distribution of diffuse in the case display region 710.Similarly, if the checkbox for the hematogenous 757 is checked, thedisplay control unit 104 displays only similar cases including the typeof lesion distribution of hematogenous in the case display region 710.

FIG. 15 is a diagram illustrating the distribution list display region750 in which the checkbox for the bronchial 753 is checked. FIG. 16 is adiagram illustrating the base screen K3 displayed when a thumbnail imagein a first column and a first row has been selected in a base screenobtained by narrowing down similar cases on the basis of the type oflesion distribution of the bronchial 753. As illustrated in FIG. 15, ifthe checkbox for the bronchial 753 is checked, the display control unit104 displays only similar cases including the type of lesiondistribution of bronchial on the case display region 710. If thethumbnail image in the first row and the first column is selected, theselected thumbnail image is displayed across the case display region 710as illustrated in FIG. 16. In the example illustrated in FIG. 16, thenumber of similar cases including the type of lesion distribution ofbronchial is 12. Therefore, the display control unit 104 displays “12results” in the retrieval result number display region 714. The displaycontrol unit 104 also displays the name of the disease to be displayedand the name of the type of lesion distribution, “bronchial”, in thedisplay condition display region 715. Because the similar cases have notbeen narrowed down on the basis of a category displayed in the diseaselist display region 730 in the example illustrated in FIG. 16, “alldiseases” is displayed in the display condition display region 715.

Similarly, if the checkbox for the segmental 752 is checked, the displaycontrol unit 104 displays only similar cases including the type oflesion distribution of segmental in the case display region 710.

FIG. 17 is a diagram illustrating the distribution list display region750 in which the checkbox for the subpleural 756 has been checked. FIG.18 is a diagram illustrating the base screen K3 displayed when athumbnail image in a first row and a first column has been selected in abase screen obtained by narrowing down similar cases on the basis of thetype of lesion distribution of the subpleural 756. As illustrated inFIG. 17, if the checkbox for the subpleural 756 is checked, the displaycontrol unit 104 displays only similar cases including the type oflesion distribution of subpleural in the case display region 710. If thethumbnail image in the first row and the first column is selected, theselected thumbnail image is displayed across the case display region 710as illustrated in FIG. 18. In the example illustrated in FIG. 18, thenumber of similar cases including the type of lesion distribution ofsubpleural is 7. Therefore, the display control unit 104 displays “7results” in the retrieval result number display region 714. The displaycontrol unit 104 also displays the name of the disease to be displayedand the name of the type of lesion distribution, “subpleural”, in thedisplay condition display region 715. Because the similar cases have notbeen narrowed down on the basis of a category displayed in the diseaselist display region 730 in the example illustrated in FIG. 18, “alldiseases” is displayed in the display condition display region 715.

FIG. 19 is a diagram illustrating the data structure of the patientinformation 1000. The patient information management unit 202 of themedical information management system 200 accumulates the patientinformation 1000 in the patient information accumulation unit 201 andmanages the patient information 1000 for each patient. In the patientinformation 1000, personal information such as the gender and age ofeach patient, clinical information such as past medical histories, andtest information regarding a blood test or the like are registered. Asillustrated in FIG. 19, the patient information 1000 includes, for eachpatient, a patient identifier (ID) 1100, a name 1200, age 1300, gender1400, a past medical history 1500, a family medical history 1600, achief complaint 1700, test information 1800, and a confirmed diagnosis1900.

The patient ID 1100 is an identifier unique to each patient. The name1200, the age 1300, the gender 1400, the past medical history 1500, thefamily medical history 1600, and the chief complaint 1700 are the name,age, gender, past medical history, family medical history, and chiefcomplaint of each patient identified by the patient ID 1100. The testinformation 1800 indicates, as illustrated in FIG. 20, informationregarding one or more medical tests that each patient has undergone inthe past.

FIG. 20 is a diagram illustrating a data structure of the testinformation 1800 illustrated in FIG. 19. The test information 1800 isinformation regarding tests conducted on each patient and created foreach medical test. The test information 1800 includes a test ID 1810, atest time 1820, a test type 1830, and test results 1840. The test ID1810 is an identifier unique to each medical test. The test time 1820indicates a time at which each test has been conducted. The test type1830 indicates the type of medical test. Types of medical test include,for example, a blood test, a respiratory function test, endoscopy, plainroentgenography, and CT scanning.

The test results 1840 refer to, in the case of a blood test, a whiteblood cell count and values of various markers such as lactatedehydrogenase (LDH) and glutamic-pyruvic transaminase (GPT). The testresults 1840 also include a diagnosis given by a doctor on the basis ofthe various markers. In the case of an imaging test such as plainroentgenography or CT scanning, the test results 1840 include pointerinformation in captured images and pointer information in a report onresults of imaging. Images captured during a test are accumulated in themedical image data accumulation unit 203 of the medical informationmanagement system 200 in a DICOM format.

If the test type 1830 is an imaging test such as plain roentgenography,CT, MRI, or positron emission tomography (PET), medical image data isaccumulated in the image forming apparatus 2000 stored in the medicalimage data accumulation unit 203 of the medical information managementsystem 200.

FIG. 21 is a diagram illustrating a data structure of the image formingapparatus 2000. The image forming apparatus 2000 includes a test ID 1810and a series ID 2100. Because a plurality of types of image capture (forexample, plain CT, contrast CT, and the like) might be performed in asingle test, a plurality of series IDs 2100 might be associated with atest ID 1810. That is, the same number of series as the number of typesof image capture are obtained.

Besides for each type of image capture, a series is obtained for eachcondition of reconfiguration of a captured image. For example, if acaptured image is reconfigured under a lung field condition and amediastinum condition, a series is obtained for each of theseconditions. In an image reconfigured under the lung field condition,blood vessels, bronchi, alveoli, and the like are emphasized. In animage reconfigured under the mediastinum condition, structures in themediastinum, such as blood vessels and lymph nodes, are emphasized.Because an image obtained in a single operation for capturing an imagecan be reconfigured under the lung field condition and the mediastinumcondition, two series in the lung field condition and two series underthe mediastinum condition are obtained if two images obtained from plainCT and contrast CT are reconfigured under the lung condition and themediastinum condition.

In the case of an imaging test such as CT or MRI, a plurality of sliceimages are obtained in a single operation for capturing an image.Therefore, a plurality of slice IDs 2200 are associated with a series ID2100. Two series IDs “CT149123” and “CT149124” are associated with atest ID “13227989” illustrated in FIG. 21, which indicates that twoseries of CT images have been obtained from an imaging test. Inaddition, a plurality of slice IDs 2200 are associated with each of theseries IDs “CT149123” and “CT149124”.

If the test type 1830 is an imaging test such as plain roentgenography,CT, MRI, or PET, for example, a diagnosis report 3000 illustrated inFIG. 22 is accumulated in the diagnosis report management unit 205 ofthe medical information management system 200. In the diagnosis report3000, a diagnosis given by a doctor in each test are registered.

FIG. 22 is a diagram illustrating a data structure of the diagnosisreport 3000. The diagnosis report 3000 includes a test ID 1810,observations 3100, and a diagnosis 3200. The test ID 1810 is the same asthe test ID 1810 illustrated in FIG. 20. Thus, the diagnosis report 3000and the test information 1800 are associated with each other. In theobservations 3100, a text indicating a doctor's observations after eachtest is registered. In the diagnosis 3200, a text indicating a doctor'sdiagnosis after each test is registered.

FIG. 23 is a diagram illustrating a data structure of the similar casedata 4000. The similar case data 4000 is referred to in order toretrieve cases similar to a diagnosis target case and created for eachsimilar case. The similar case data 4000 is an example of additionalinformation regarding similar cases. The similar case data 4000 isaccumulated in the similar case data accumulation unit 301 of the caseretrieval system 300 for each similar case. As illustrated in FIG. 23,the similar case data 4000 includes a similar case ID 4100, a slice ID4200, ROI information 4300, image feature data 4400, thumbnail imagedata 4500, lesion distribution information 4600, a confirmed diagnosis(large category) 4700, and a confirmed diagnosis (small category) 4800.

The similar case ID 4100 is an identifier of the similar case data 4000.Here, because the similar case data 4000 is generated for each ROI setin a slice image of a similar case, the similar case ID 4100 is, in asense, an identifier of each ROI. In the example illustrated in FIG. 23,the similar case ID 4100 is a character string including “SIM” andsubsequent numbers.

The slice ID 4200 is an identifier of a slice image in which a ROI isset and the same as the slice ID 2200 illustrated in FIG. 21. The ROIinformation 4300 is information indicating a position of a ROI set in aslice image. FIG. 24 is a diagram schematically illustrating a ROI setin a slice image 3101. In the example illustrated in FIG. 24, the setROI is rectangular. Therefore, the ROI information 4300 includes fourvalues, namely coordinates (xl, yt) of an upper-left corner of the ROIand coordinates (xr, yb) of a lower-right corner of the ROI. Obviously,the ROI need not be rectangular. In that case, a parameter for uniquelyidentifying the ROI is used as the ROI information 4300. For example, ifthe ROI is circular, coordinates of the center of the circle and theradius of the circle are used as the ROI information 4300.

The image feature data 4400 is a feature value of certain dimensions (Ndimensions in this embodiment) extracted from the ROI defined by the ROIinformation 4300. The thumbnail image data 4500 is image data regardinga thumbnail image to be displayed in the case display region 710generated on the basis the slice image in the DICOM format identified bythe slice ID 4200. Here, in the thumbnail image data 4500, for example,pixel values of a thumbnail image are arranged in order of rasterscanning, that is, from an upper-left corner to a lower-right corner ofthe thumbnail image. As described above, a DICOM image obtained as aresult of a CT test is a 11-bit (pixel value: −1,000 to +1,000) imagehaving 512×512 pixels. Therefore, in this embodiment, in order topromptly display a thumbnail image, a thumbnail image having 8-bit pixelvalues is created in advance by performing a low-resolution process anda tone conversion process on an original DICOM image and registered inthe similar case data 4000. For example, the medical informationmanagement system 200 may create a thumbnail image and transmit thethumbnail image to the case retrieval system 300, or the case retrievalsystem 300 may create a thumbnail image after obtaining a DICOM imagefrom the medical information management system 200.

The lesion distribution information 4600 is a distribution flag value (1for applicable and 0 for inapplicable) indicating whether a targetsimilar case corresponds to each of predetermined types of lesiondistribution, namely diffuse 4610 to hematogenous 4670.

The confirmed diagnosis (large category) 4700 is a large category intowhich a target similar case falls. The confirmed diagnosis (largecategory) 4700 is used for narrowing down similar cases on the basis oflarge categories.

The confirmed diagnosis (small category) 4800 is a small category intowhich a target similar case falls. The confirmed diagnosis (smallcategory) 4800 is used for narrowing down similar cases on the basis ofsmall categories.

Large categories uniquely corresponding to small categories are definedin advance, and the confirmed diagnosis (large category) 4700 and theconfirmed diagnosis (small category) 4800 are stored in the similar casedata 4000 in accordance with the corresponding relationship.

The medical image data accumulation unit 203 identifies a series ID 2100from a slice ID 2200 illustrated in FIG. 21 for the confirmed diagnosis(small category) 4800. The patient information accumulation unit 201identifies a test ID 1810 from the identified series ID 2100, patientinformation 1000 (FIG. 19) from the identified test ID 1810, and aconfirmed diagnosis 1900 from the identified patient information 1000for each patient.

Next, a procedure from a beginning of a reading operation to a beginningof retrieval of similar cases performed by the information terminal 100in cooperation with the medical information management system 200 andthe case retrieval system 300 will be described.

FIG. 25 is a sequence diagram illustrating a process performed until thecase retrieval system 300 receives, from the information terminal 100, arequest to retrieve similar cases after the information terminal 100requests the reading target test list from the medical informationmanagement system 200. In FIG. 25, rectangles illustrated in a left partof the sequence diagram indicate screens displayed on the display 101 asa result of processing in corresponding steps. In FIG. 25, “A” in theinformation terminal 100 indicates the medical information managementapplication, and “B” indicates the similar case retrieval application.It is assumed that the medical information management application isactivated in advance before this sequence begins.

First, the information terminal 100 receives, from the user (a doctorwho reads a medical image), a request to display the reading target testlist through the reading target test list button 701 (FIG. 4). Theinformation terminal 100 transmits the request to display the readingtarget test list to the communication control unit 206 of the medicalinformation management system 200 through the input control unit 103 andthe communication control unit 110 (S510).

The patient information management unit 202 of the medical informationmanagement system 200 generates a reading target test list, which is alist of tests for which imaging tests have been conducted but readinghas not been performed. The patient information management unit 202transmits the generated reading target test list to the communicationcontrol unit 110 of the information terminal 100 through thecommunication control unit 206 (S520). Here, the reading target testlist includes the patient information 1000 and the test information 1800regarding target patients.

The display control unit 104 of the information terminal 100 displaysthe reading target test list received by the communication control unit110 on the display 101 (S530).

FIG. 26 is a diagram illustrating an example of a display screen K5 forthe reading target test list. The reading target test list includes apatient list display region 800 in which patients corresponding to thetests for which reading has not been performed are displayed and a testlist display region 810 in which information regarding series includedin a test is displayed. The patient list display region 800 includesfields of “patient ID”, “patient name”, “test time”, “test ID”, and“test type”. In the fields of “patient ID” and “patient name”, thepatient ID 1100 and the name 1200, respectively, registered in thepatient information 1000 are displayed, and in the fields of “testtime”, “test ID”, and “test type”, the test time 1820, the test ID 1810,and the test type 1830, respectively, registered in the test information1800 are displayed. The test list display region 810 is a region fordisplaying details of a test selected by the user in the patient listdisplay region 800 and includes fields of “series ID”, “definition”, and“image”. Here, because the user has not selected any test (correspondsto rows) in the patient list display region 800, no information isdisplayed in the test list display region 810.

The user selects one of the tests displayed in the patient list displayregion 800 as a test to be read. The input control unit 103 detects theselection, and, as illustrated in FIG. 25, the communication controlunit 110 transmits a request to display all series included in a test IDof the selected test to the medical information management system 200(S540).

After the communication control unit 206 of the medical informationmanagement system 200 receives the display request, the patientinformation management unit 202 refers to the image forming apparatus2000 illustrated in FIG. 21. The patient information management unit 202obtains all slice images of all the series included in the test IDspecified by the display request and transmits the slice images to theinformation terminal 100 through the communication control unit 206(S550). For example, in the example illustrated in FIG. 21, if the userselects a test identified from the test ID “13227989”, all slice imagesincluded in the series identified from the series IDs “CT149123” and“CT149124” are transmitted in S550.

If the communication control unit 110 of the information terminal 100obtains images of the all the series, the display control unit 104displays a series list including information regarding all the seriesincluded in the specified test ID in the test list display region 810(S560).

FIG. 27 is a diagram illustrating an example of the display screen K5for the reading target test list after a test is selected. In thepatient list display region 800 illustrated in FIG. 27, a selected rowis highlighted. In the example illustrated in FIG. 27, a second row,which indicates a test conducted on “Panataro”, is selected in thepatient list display region 800. Therefore, series IDs, definitions, andimages of the selected test are displayed in the test list displayregion 810. In the field of “series ID”, the series IDs associated witha test ID of the selected test are displayed, and in the field of“image”, a thumbnail image of a slice image representing each of thedisplayed series IDs is displayed. Here, an image at a certain sliceposition is adopted as the slice image representing each series ID. Thecertain slice position may be a top slice position or may be a centralslice position. Each definition indicates image capture conditions,reconfiguration conditions, and the like of each series. Although notillustrated, for example, each definition is registered and associatedwith a series ID in the image forming apparatus 2000 illustrated in FIG.21.

In FIG. 25, if the user selects a series to be read in the test listdisplay region 810 and the input control unit 103 detects the selection,the display control unit 104 displays a first slice image of theselected series in the medical image viewer 705 as illustrated in FIG. 4(S570). As described above with reference to FIG. 4, the display controlunit 104 displays all slice images of the selected series on the display101 such that each slice image can be displayed through scrolling. Theuser performs image diagnosis while performing an operation fordisplaying each slice image through scrolling. If it is difficult forthe user to perform image diagnosis, the user activates the similar caseretrieval application.

Before activating the similar case retrieval application, the user tapsthe ROI input button 702 (FIG. 4) with an object (for example, theuser's finger) to activate a ROI input mode. After the ROI input mode isactivated, the ROI management unit 105 begins to manage the informationterminal 100, and the information terminal 100 waits for an input of aROI.

The user sets, through the operation unit 102, a ROI including a lesionin a slice image displayed on the medical image viewer 705 of thedisplay 101 (S580). As illustrated in FIG. 24, for example, the usertaps the display 101 with an object to input the coordinates of theupper-left corner of the ROI in the slice image 3101. The user thendrags the object on the display 101 in a lower-right direction andreleases the object from the display 101 to input the coordinates of thelower-right corner of the ROI.

After the input control unit 103 detects the operation for setting aROI, the ROI management unit 105 receives data regarding the coordinatesof the upper-left corner and the lower-right corner of the ROI from theinput control unit 103 and generates the received data as ROIinformation.

Here, the user activates the similar case retrieval application bytapping the similar case retrieval button 703 with the object. As aresult, the ROI management unit 105 transmits the generated ROIinformation to the communication control unit 110 (S590).

Simultaneously, the ROI management unit 105 transmits a slice image ofthe diagnosis target case to the communication control unit 110 (S600).In this case, among all the slice images of the series received in S550by the information terminal 100 from the medical information managementsystem 200, the slice image (retrieval query image) in which the userhas set a ROI in the series that the user has selected is transmitted.

Next, the communication control unit 110 receives the ROI informationtransmitted from the ROI management unit 105 and transmits the ROI tothe communication control unit 304 of the case retrieval system 300(S601).

Simultaneously, the communication control unit 110 receives the sliceimage transmitted from the ROI management unit 105 and transmits theslice image to the communication control unit 304 of the case retrievalsystem 300 (S602).

Although the slice image is transmitted in S600 and S601, only a sliceID of the slice image may be transmitted, instead. In this case, afterreceiving the slice ID, the case retrieval system 300 may obtain theslice image from the medical information management system 200 byspecifying the slice ID.

Next, a process performed until the information terminal 100 initiallydisplays results of retrieval of similar cases after the case retrievalsystem 300 performs the retrieval of similar cases will be described.

FIG. 28 is a sequence diagram illustrating the process performed untilthe information terminal 100 displays a base screen on the basis ofresults of retrieval of similar cases received from the case retrievalsystem 300 after the case retrieval system 300 receives a request toperform the retrieval of similar cases from the information terminal100.

The image feature extraction unit 302 of the case retrieval system 300extracts a predetermined image feature of a plurality of dimensions setin a retrieval query image (S610).

As the image feature, an image feature relating to the shape of an organor a lesion in a medical image, an image feature relating to luminancedistribution, or the like may be adopted. For example, in Nemoto,Shimizu, Hagiwara, Kobatake, and Nawano “Improvement of Tumor DetectionPerformance in Mammograms by Feature Selection from a Large Number ofFeatures and Proposal of Fast Feature Selection Method”, Systems andComputers in Japan, D-II, Vol. J88-D-II, No. 2, pp. 416-426, February2005, use of an image feature of 490 dimensions is described. In thisembodiment, for example, the image feature described in this non-patentliterature is used. This, however, is just an example, and another imagefeature may be used, instead.

The similar case retrieval unit 303 compares the image feature extractedby the image feature extraction unit 302 with the image features ofsimilar cases accumulated in the similar case data accumulation unit 301(S620). Here, the similar case retrieval unit 303 compares the imagefeatures with each other by calculating a distance between image featuredata extracted from the retrieval query image and the image feature data4400 registered in the similar case data 4000 (FIG. 23) accumulated inthe similar case data accumulation unit 301 for each similar case.

Next, the similar case retrieval unit 303 sorts similar cases whosedistances are smaller than a certain threshold in order of ascendingdistance and determines the similar cases as similar cases to betransmitted (S630). Next, the communication control unit 304 transmits,among pieces of data included in the similar case data 4000 accumulatedin the similar case data accumulation unit 301, similar case IDs 4100,slice IDs 4200, ROI information 4300, thumbnail image data 4500, lesiondistribution information 4600, confirmed diagnoses (large category)4700, confirmed diagnoses (small category) 4800, and the distances,which have been calculated by the similar case retrieval unit 303, ofthe similar cases determined as transmission targets to the informationterminal 100 (S640).

Next, the display control unit 104 generates, on the basis of thetransmitted information, the initial base screen K2 (FIG. 5) in whichresults of the retrieval of similar cases are displayed (S650).

FIG. 29 is a flowchart illustrating details of a process for generatingthe initial base screen K2 indicated by S650 illustrated in FIG. 28.

First, in S1000, the display control unit 104 counts the number ofsimilar cases received in S640 illustrated in FIG. 28 and displays thenumber in the retrieval result number display region 714.

Next, in S1100, the display control unit 104 displays “all diseases” inthe display condition display region 715. The reason why “all diseases”are displayed is that, in the initial base screen K2, the user has notnarrowed down the similar cases using a category or a type of lesiondistribution.

Next, in S1200, the display control unit 104 displays, in the casedisplay region 710, thumbnail images of a certain number of similarcases that have been received in S640 illustrated in FIG. 28 and thatcan be displayed in the case display region 710. The display controlunit 104 also displays a confirmed diagnosis and similarity for eachthumbnail image.

In the example illustrated in FIG. 5, the maximum number of similarcases that can be displayed in the case display region 710 is 20. Themaximum number is determined in advance. Alternatively, the user may beable to change the maximum number arbitrarily. If the number of similarcases received in S640 illustrated in FIG. 28 is larger than the maximumnumber, the display control unit 104 displays the vertically long scrollbar 716 (FIG. 5) at a right end of the case display region 710. As aresult, the user can view thumbnail images of similar cases that are notdisplayed in the initial base screen K2 by moving the scroll bar 716.

Next, in S1300, a disease list is generated and displayed. First, thedisease list is generated from the similar cases received in S640illustrated in FIG. 28. The disease list is a list in which the similarcases received in S640 are classified using confirmed diagnoses.

Here, it is assumed that the number of similar cases received in S640 isNC. The disease list management unit 108 generates the disease listusing the confirmed diagnosis (large category) 4700 and the confirmeddiagnosis (small category) 4800 registered in the similar case data 4000regarding each of the NC similar cases. The disease list management unit108 manages the generated disease list as data having a table format asillustrated in FIG. 30.

FIG. 30 is a diagram illustrating a data structure of the disease listgenerated in S1300 illustrated in FIG. 29. The disease list includesfields of “disease ID”, “large category”, “small category”, “number”,and “similar case ID”. “Disease ID” is an identifier provided for eachconfirmed diagnosis. Here, a disease ID is provided for each combinationof a large category and a small category.

“Large category” is a confirmed diagnosis indicated by the confirmeddiagnosis (large category) 4700 registered in the similar case data4000. “Small category” is a confirmed diagnosis indicated by theconfirmed diagnosis (small category) 4800 registered in the similar casedata 4000. “Number” is the number of similar cases corresponding to theconfirmed diagnosis indicated by “disease ID”. “Similar case ID” is asimilar case ID indicating a case similar to the disease indicated by“disease ID”.

The disease list management unit 108 extracts the confirmed diagnosis(large category) 4700 and the confirmed diagnosis (small category) 4800for each of all the pieces of similar case data 4000 received in S640and classifies, as similar cases whose confirmed diagnoses are the same,pieces of similar case data 4000 whose confirmed diagnoses (largecategory) 4700 and confirmed diagnoses (small category) 4800 are thesame as one another. The disease list management unit 108 then countsthe number of similar cases whose confirmed diagnoses are the same andregisters the number to the field of “number” in a record of acorresponding confirmed diagnosis. The disease list management unit 108also registers similar case IDs of the similar cases classified into thesame confirmed diagnosis to the field of “similar case ID” of the recordof the corresponding confirmed diagnosis.

In the example illustrated in FIG. 30, a disease ID “DIS528” is providedfor a confirmed diagnosis whose large category is “neoplastic” and smallcategory is “primary lung cancer”. Because the number of similar casescorresponding to this confirmed diagnosis is 10, 10 is registered to thefield of “number” of the corresponding record, and similar case IDs“SIM258”, “SIM551”, “SIM1209”, “SIM2341”, and the like of similar casescorresponding to this confirmed diagnosis are registered to the field of“similar case ID” of the corresponding record.

The display control unit 104 generates the disease list display region730 using the disease list generated in this manner and displays thedisease list display region 730 on the display 101.

FIGS. 31, 32, and 33 are diagrams illustrating a first display example,a second display example, and a third display example, respectively, ofthe disease list display region 730. As illustrated in FIG. 31, in thefirst display example, similar cases obtained as a result of retrievalof similar cases are displayed in a list in descending order in terms ofsmall categories.

As illustrated in FIG. 32, in the second display example, similar casesobtained as a result of retrieval of similar cases are displayed in alist in descending order in terms of large categories.

As illustrated in FIG. 33, in the third display example, similar casesobtained as a result of retrieval of similar cases are displayed as alist in descending order in terms of large categories and in descendingorder in terms of small categories included in each large category. Inthis case, confirmed diagnoses are represented in a hierarchicalstructure of large categories and small categories.

FIG. 34 is a diagram illustrating switching of the disease list displayregion 730 illustrated in FIG. 32. As indicated in an upper part of FIG.34, if the input control unit 103 detects an operation for selecting oneof the large categories displayed in the list, the display control unit104 displays, as indicated in a lower part of FIG. 34, small categoriesbelonging to the selected large category in descending order. Here, theuser may select one of the large categories displayed in the list in thedisease list display region 730 by, for example, double-tapping ortapping a desired large category. In the example illustrated in FIG. 34,since “nonneoplastic” has been double-tapped, small categories belongingto “nonneoplastic” is displayed in the list.

In the lower part of FIG. 34, if the user double-taps or taps a regionin which the small categories are displayed, the display control unit104 may hide the small categories displayed in the region.

The display control unit 104 may determine small categories belonging toa large category by referring to the disease list (FIG. 30). Forexample, in the example illustrated in FIG. 30, since aspergillosis andcryptococcosis are associated with mycosis, the display control unit 104may determine that aspergillosis and cryptococcosis belong to mycosis.

In FIG. 29, in S1400, a distribution list is generated and displayed.First, a distribution list is generated from the similar cases receivedin S640. The distribution list is a list in which the similar casesreceived in S640 are classified using types of lesion distribution.

The disease list management unit 108 generates the distribution listusing the lesion distribution information 4600 registered in each of theNC pieces of similar case data 4000. As illustrated in FIG. 35, thedistribution list management unit 109 manages the generated distributionlist as data having a table format.

FIG. 35 is a diagram illustrating a data structure of the distributionlist generated in S1400 illustrated in FIG. 29. The distribution listincludes fields of “type of distribution”, “number of cases”, and“similar case ID”. “Type of distribution” indicates a plurality ofpredetermined types of lesion distribution, such as diffuse andsegmental. “Number of cases” indicates the number of similar casescorresponding to each type of lesion distribution. “Similar case ID”indicates a similar case ID indicating a similar case corresponding toeach type of lesion distribution.

The distribution list management unit 109 extracts the lesiondistribution information 4600 for each of all the pieces of similar casedata 4000 received in S640. The distribution list management unit 109then counts the number of types of lesion distribution for which 1(applicable) is set to distribution flag values and registers the numberto the field of “number of cases” of a record of corresponding types oflesion distribution. The distribution list management unit 109 alsoregisters similar case IDs of similar cases for which 1 is set todistribution flag values to the field of “similar case ID” of the recordof the corresponding types of lesion distribution.

In the example illustrated in FIG. 35, since the number of similar casescorresponding to “diffuse” is 3, 3 is registered to the field of “numberof cases” of a record of “diffuse”. In addition, similar case IDs“SIM2521”, “SIM4123”, and “SIM5225” of the similar cases correspondingto “diffuse” are registered to the field of “similar case ID”.

The display control unit 104 generates the distribution list displayregion 750 using the distribution list generated in this manner anddisplays the distribution list display region 750 on the display 101.

The distribution list display region 750 generated using thedistribution list illustrated in FIG. 35 is illustrated in FIG. 12. InFIG. 35, since the number of cases corresponding to “segmental” is 0,the segmental 752 illustrated in FIG. 12 is displayed in the inactivestate. Since the number of similar cases of the other types of lesiondistribution is 1 or more, these types of lesion distribution aredisplayed in the active state.

In FIG. 29, in S1500, the display control unit 104 displays, asillustrated in FIG. 5, the base screen K2 in which thumbnail images ofthe similar cases are displayed in the case display region 710 on thedisplay 101.

In FIG. 28, in S651, one of the similar cases is selected by, forexample, double-tapping one of the thumbnail images displayed in thecase display region 710. In S652, the display control unit 104 displays,on the display 101, the base screen K3 (FIG. 6A) in which the thumbnailimage of the selected similar case across the case display region 710. Asimilar case may be selected through an operation other than tapping ordouble tapping.

Next, processes performed by the information terminal 100, the medicalinformation management system 200, and the case retrieval system 300when the sequence diagrams of FIGS. 25 and 28 are seen at a level of theapplications will be described.

FIG. 36 is a sequence diagram at a time when the sequence diagrams ofFIGS. 25 and 28 are seen at the level of the applications. In FIG. 36,the same steps as those illustrated in FIGS. 25 and 28 are given thesame reference numerals.

In FIG. 36, “A” indicates a process performed by the medical informationmanagement application executed by the information terminal 100, and “B”indicates a process performed by the similar case retrieval applicationexecuted by the information terminal 100. In the following description,the medical information management application will be referred to as“Application A”, and the similar case retrieval application will bereferred to as “Application B”.

First, Application A receives a request to display a reading target testlist from the user and transmits the request to the medical informationmanagement system 200 (S510). Upon receiving the request to display atest list, the medical information management system 200 makes a list oftests for which imaging tests have been conducted but reading has notbeen performed. The medical information management system 200 thengenerates a test list to be read and transmits the test list toApplication A.

Upon receiving the reading target test list, Application A displays thereading target test list on the display 101 as illustrated in FIG. 26.If the user selects one of the tests included in the reading target testlist (S530), Application A transmits a request to display the selectedtest to the medical information management system 200 (S540).

Upon receiving the request to display the test, the medical informationmanagement system 200 transmits all slice images of all series includedin a test ID specified by the request to display the test to ApplicationA (S550).

Next, as illustrated in FIG. 27, Application A displays a series listfor displaying information relating to all the series included in thespecified test ID (S560).

Next, if the user selects a series to be read from the series list,Application A displays a slice image at a first slice position of theselected series on the distribution list display region 705 asillustrated in FIG. 4 (S570). At this time, the user displays a desiredslice image on the medical image viewer 705 by performing an operationfor displaying each slice image through scrolling.

Next, Application A receives, from the user, an operation for setting aROI in the slice image displayed on the medical image viewer 705 (S580).

Next, Application A generates ROI information indicating the ROI set bythe user and transmits the ROI information to Application B along withthe slice image (slice image of a diagnosis target case) in which theROI has been set (S590 and S600).

Next, upon receiving the slice image of a diagnosis target case and theROI information, Application B transmits the slice image and the ROIinformation to the case retrieval system 300 (S601 and S602).

Upon receiving the slice image and the ROI information, the caseretrieval system 300 performs the processing in S610 to S640 as in FIG.28.

Next, Application B generates the initial base screen K2 using thesimilar case data transmitted in S640 as illustrated in FIG. 5 (S650).Application B then performs the processing in S651 and S652 as in FIG.28.

Although an example in which the case retrieval system 300 extracts animage feature has been described above, the information terminal 100 mayextract an image feature, instead. FIG. 37 is a block diagramillustrating the information terminal 100, the medical informationmanagement system 200, and the case retrieval system 300 at a time whena mode in which the case retrieval system 300 extracts an image featureis adopted.

Differences from FIG. 2 are that an image feature extraction unit 113 isadded to the information terminal 100 and that the image featureextraction unit 302 is omitted from the case retrieval system 300.

FIG. 38 is a sequence diagram illustrating a process performed in theconfiguration illustrated in FIG. 37 until the information terminal 100displays a base screen on the basis of results of retrieval of similarcases received from the case retrieval system 300 after the informationterminal 100 sets a ROI.

A difference from FIGS. 25 and 28 is that after the ROI management unit105 performs the process for transmitting a slice image of a diagnosistarget case to the communication control unit 110 (S600), theinformation terminal 100 extracts an image feature (S603) and transmitsthe extracted image to the case retrieval system 300 (S604). The processfor extracting an image feature (S604) is the same as that performed bythe case retrieval system 300.

FIG. 39 is a sequence diagram at a time when the process performed inthe configuration illustrated in FIG. 37 until the information terminal100 displays a base screen on the basis of results of retrieval ofsimilar cases received from the case retrieval system 300 afterrequesting the reading target test list from the medical informationmanagement system 200 is seen at the level of the applications. In FIG.39, the same steps as those illustrated in FIGS. 25, 28, and 36 aregiven the same reference numerals.

In FIG. 39, “A” indicates a process performed by the medical informationmanagement application executed by the information terminal 100, and “B”indicates a process performed by the similar case retrieval applicationexecuted by the information terminal 100. In the following description,the medical information management application will be referred to as“Application A”, and the similar case retrieval application will bereferred to as “Application B”.

Differences from FIG. 36 are in S603 and S604. In FIG. 39, theinformation terminal 100 extracts an image feature. Therefore,Application B extracts an image feature from a ROI set in a slice imageof a diagnosis target case (S603) and transmits the extracted imagefeature to the case retrieval system 300 (S604).

Next, a process for switching a similar case displayed on the display101 performed by the user by performing a swipe operation on the display101 of the information terminal 100 will be described.

FIG. 40 is a sequence diagram illustrating the process for switching asimilar case displayed on the display 101 of the information terminal100 on the basis of a swipe operation. FIG. 41 is a flowchartillustrating the process corresponding to the sequence diagram of FIG.40. FIGS. 42 to 47 are diagrams illustrating the base screen K3 in whicha thumbnail image displayed in the case display region 710 is switchedby a swipe operation performed by the user.

At a beginning of the process illustrated in FIGS. 40 and 41, thedisplay 101 of the information terminal 100 displays the base screen K3illustrated in FIG. 6A. The thumbnail image in the first row and thefirst column (that is, the thumbnail image of a similar case having thehighest similarity) in the case display region 710 illustrated in FIG. 5is displayed across the case display region 710 of the base screen K3illustrated in FIG. 6A.

In S700 illustrated in FIGS. 40 and 41, first, the input control unit103 of the information terminal 100 detects a swipe operation performedby the user. The input control unit 103 notifies the image generationunit 112 of the swipe operation.

Next, in S710, the image generation unit 112 generates a thumbnail image(that is, a thumbnail image in the first row and a second column in thecase display region 710 illustrated in FIG. 5) of a similar case havingsecond highest similarity on the basis of the notification from theinput control unit 103. The image generation unit 112 outputs thegenerated thumbnail image to the display control unit 104.

Next, in S720, the image generation unit 112 generates, as illustratedin FIGS. 42 and 43, an intermediate image, in which the thumbnail imagebefore switching of the similar case and the thumbnail image after theswitching are arranged adjacent to each other, during the swipeoperation performed with an object 350 (the user's finger) so that theintermediate image appears in accordance with the swipe operation. Theimage generation unit 112 outputs the generated intermediate image tothe display control unit 104. The display control unit 104 displays theintermediate image input from the image generation unit 112 in the casedisplay region 710.

Next, in S730, after the swipe operation is completed, the displaycontrol unit 104 displays, as illustrated in FIG. 44, the thumbnailimage of the similar case having the second highest similarity inputfrom the image generation unit 112 in S710 across the case displayregion 710.

If the swipe operation is further performed with the base screen K3illustrated in FIG. 44 displayed on the display 101, the display controlunit 104 displays, as illustrated in FIGS. 45 and 46, an intermediateimage in the case display region 710 in S720 after the processing inS700 and S710, which have been described above. Next, in S730, after theswipe operation is completed, the display control unit 104 displays, asillustrated in FIG. 47, a thumbnail image (a thumbnail image in thefirst row and a third column in the case display region 710 illustratedin FIG. 5) of a similar case having third highest similarity across thecase display region 710.

In the first embodiment, as illustrated in FIGS. 44 and 47, the displaycontrol unit 104 displays a thumbnail image while adjusting a positionof the thumbnail image such that the center of a CROI matches the centerof the case display region 710 during the swipe operation. Asillustrated in FIGS. 44 and 47, the display control unit 104 displays abackground image in a region 710B, which is a region inside the casedisplay region 710 but outside the thumbnail image. The background imagemay be grey, or may be white or black. Thus, the region 710B can beidentified. In the following description, details of a process forgenerating a thumbnail image illustrated in FIG. 44 or 47 performed bythe image generation unit 112 will be described with reference to FIG.48.

FIG. 48 is a block diagram illustrating a detailed configuration of theimage generation unit 112. The image generation unit 112 includes aninput determination section 1121, a display case determination section1122, a display region determination section 1123, a display imagegeneration section 1124, and an elapsed time determination section 1125.The display 101 of the information terminal 100 displays the base screenK3 illustrated in FIG. 6A.

If the input control unit 103 detects an operation performed by the userusing the operation unit 102, the input control unit 103 outputs aresult of the detection to the input determination section 1121. Theinput determination section 1121 determines whether the operationperformed by the user using the operation unit 102 is a swipe operationon the basis of the result of the detection input from the input controlunit 103. If the input determination section 1121 determines that theoperation is a swipe operation, the input determination section 1121outputs information indicating that the operation is a swipe operationand a direction of the swipe operation (for example, leftward orrightward) to the display case determination section 1122.

The display case determination section 1122 refers to the similar caseID 4100 (FIG. 23) and the distance calculated by the similar caseretrieval unit 303 corresponding to a similar case displayed in the casedisplay region 710. If the direction of the swipe operation input fromthe input determination section 1121 is leftward, the display casedetermination section 1122 obtains a similar case ID of a similar casewhose having shortest distance next to that of the similar casecurrently displayed in the case display region 710 (that is, a similarcase having highest similarity next to that of the current similarcase).

On the other hand, if the direction of the swipe operation input fromthe input determination section 1121 is rightward, the display casedetermination section 1122 obtains a similar case ID of a similar casehaving longest distance next to that of the similar case currentlydisplayed in the case display region 710 (that is, a similar case havinghigher similarity than the current similar case).

The display case determination section 1122 determines a similar casecorresponding to the obtained similar case ID as a similar case to bedisplayed next. If the display case determination section 1122 does notobtain a similar case ID, the display case determination section 1122determines the currently displayed similar case as the similar case tobe displayed next. That is, the similar case displayed in the casedisplay region 710 remains the same. For example, even if a rightwardswipe operation is performed with the base screen K3 illustrated in FIG.6A displayed, the display screen does not change because the similarityof the similar case currently displayed in the case display region 710is the highest.

The display region determination section 1123 refers to the ROIinformation 4300 (FIG. 23) of the similar case data 4000 regarding thesimilar case determined by the display case determination section 1122.The display region determination section 1123 determines coordinates ofa display region of a thumbnail image of the similar case such that thecenter coordinates of the CROI match the center of the case displayregion 710.

FIG. 49 is a diagram schematically illustrating the coordinates of thedisplay region determined by the display region determination section1123. Coordinates (xdl, ydt) of an upper-left corner of the displayregion and coordinates (xdr, ydb) of a lower-right corner of the displayregion illustrated in FIG. 49 can be calculated using the followingexpression using center coordinates (xc, yc) of the CROI and a verticaldimension h and a horizontal direction w of the display region.

xdl=xc−w/2

ydt=yc−h/2

xdr=xc+w/2

ydb=yc+h/2

At this time, if the coordinates (xdl, ydt) or (xdr, ydb) of the displayregion are outside the original thumbnail image (for example, negativevalues), the display region may be displayed, as described above, inblack as a background image whose pixel values are 0. Alternatively, theuser may specify color. In FIG. 49, the coordinates (xdl, ydt) of theupper-left corner are outside the original thumbnail image, and thebackground image is displayed in the region 710B.

The display image generation section 1124 generates an image to bedisplayed in the case display region 710 on the basis of the coordinatesof the display region determined by the display region determinationsection 1123 and the thumbnail image of the similar case determined bythe display case determination section 1122. The display imagegeneration section 1124 outputs the generated image to the displaycontrol unit 104.

As a result of the above process, even if a thumbnail image displayed inthe case display region 710 is switched to a next thumbnail image, thenext thumbnail image is displayed such that the CROI is located at thecenter of the case display region 710.

The position of the CROI of each similar case is usually different.Therefore, if a thumbnail image is switched without making anyadjustment, a doctor needs to visually search for the CROI. When thereare many images, this operation might be physically exhausting.

On the other hand, according to the first embodiment, the doctor doesnot need to visually search for the CROI each time a new thumbnail imageof a similar case is displayed. Therefore, the doctor can observe theCROI immediately. As a result, when thumbnail images of a plurality ofsimilar cases are sequentially observed through a swipe operation, theobservation can be less exhausting and a reading efficiency improves,thereby improving an efficiency of diagnosis.

Next, a process when the user has not performed a swipe operation for acertain period of time after switching a thumbnail image of a similarcase displayed in the case display region 710 by performing a swipeoperation on the display 101 of the information terminal 100 will bedescribed with reference to FIGS. 48 and 50.

FIG. 50 is a sequence diagram illustrating a process performed when theuser has not performed a swipe operation for a certain period of timeafter switching a thumbnail image of a similar case by performing aswipe operation on the information terminal 100.

In FIG. 48, the input control unit 103 outputs a detection signal to theelapsed time determination section 1125 each time the input control unit103 detects a swipe operation. The elapsed time determination section1125 measures time elapsed since a last swipe operation performed by theuser on the basis of the detection signal from the input control unit103. If the elapsed time exceeds a predetermined threshold (for example,10 seconds), the elapsed time determination section 1125 outputs anelapsed signal indicating the excess to the display region determinationsection 1123.

In S740 illustrated in FIG. 50, if the elapsed time determinationsection 1125 outputs the elapsed signal to the display regiondetermination section 1123, the display region determination section1123 begins a process for changing the display region of a thumbnailimage displayed in the case display region 710. The image generationunit 1123 determines the display region of a thumbnail image such thatthe center of the original thumbnail image matches the center of thecase display region 710.

Next, in S750, the display image generation section 1124 generates theoriginal thumbnail image on the basis of the display region determinedby the display region determination section 1123. The display imagegeneration section 1124 outputs the generated original thumbnail imageto the display control unit 104.

Next, in S760, the display control unit 104 displays the thumbnail imagegenerated by the elapsed time determination section 1125 in the casedisplay region 710 of the display 101.

FIGS. 51 and 52 are diagrams illustrating examples of the base screen K3displayed on the display 101 in the sequence illustrated in FIG. 50.

In S740 illustrated in FIG. 50, the base screen K3 illustrated in FIG.47 after the last swipe operation is completed is displayed on thedisplay 101.

If the time elapsed since the last swipe operation exceeds the threshold(for example, 10 seconds) without another swipe operation, the basescreen K3 illustrated in FIG. 51, in which the display region isdetermined such that the center of the original thumbnail image matchesthe center of the case display region 710, is displayed on the display101.

If the center position is suddenly changed from that in the base screenK3 illustrated in FIG. 47 to that in the base screen K3 illustrated inFIG. 51, the doctor might need to search for the CROI, which isundesirable. Therefore, if the center position is reset to that in theoriginal thumbnail image, the image generation unit 112 may generate, asillustrated in FIG. 52, a thumbnail image whose display region issomewhere between that of the thumbnail image illustrated in FIG. 47 andthat of the thumbnail image illustrated in FIG. 51. The display controlunit 104 may then display the generated thumbnail image on the display101. Alternatively, not only the thumbnail image illustrated in FIG. 52but also a plurality of thumbnail images may be displayed between thestate illustrated in FIG. 47 and the state illustrated in FIG. 51. Inthis case, the processing from S740 to S760 illustrated in FIG. 10 maybe repeated a plurality of times to generate a plurality of thumbnailimages. As a result, the center position of a thumbnail image can besmoothly changed.

When a swipe operation is not performed, the user might be closelylooking at a current image. Therefore, the user might desire to checkthe entirety of the image. For this reason, in the first embodiment, asdescribed with reference to FIG. 47 and FIGS. 50 to 52, if the timeelapsed since the last swipe operation exceeds the threshold (forexample, 10 seconds), the base screen K3 in which the center of theoriginal thumbnail image matches the center pf the case display region710 is displayed on the display 101. As a result, the number ofoperations performed to view the entirety of a thumbnail image can bedecreased.

Second Embodiment

Next, a second embodiment will be described. In the second embodiment,the user enlarges a thumbnail image of a similar case displayed on theinformation terminal 100 and then performs a swipe operation to switchthe displayed thumbnail image of the similar case.

FIG. 53 is a sequence diagram illustrating a process performed until athumbnail image of a next similar case is displayed after the userperforms an enlargement operation. FIG. 54 is a flowchart illustratingthe process corresponding to the sequence diagram of FIG. 53. FIGS. 55and 56 are diagrams illustrating a pinch-out operation performed by theuser on a thumbnail image. FIGS. 57 to 59 are diagrams illustrating thebase screen K3 in which the thumbnail image displayed in the casedisplay region 710 is switched through a swipe operation performed bythe user. In FIG. 54, the same steps as those illustrated in FIG. 53 aregiven the same reference numerals.

As illustrated in FIGS. 55 and 56, the input control unit 103 of theinformation terminal 100 detects an enlargement operation (a pinch-outoperation in FIGS. 55 and 56) performed by the user (S800). Asillustrated in FIGS. 55 and 56, the user performs the pinch-outoperation using an object 350 (for example, two fingers of the user).The enlargement operation may be an operation other than the pinch-outoperation, that is, for example, a tap operation, instead.

Next, the image generation unit 112 enlarges the thumbnail image of thesimilar case by an enlargement ratio according to the amount of thepinch-out operation performed by the user (S810). As illustrated in FIG.56, the display control unit 104 displays an enlarged thumbnail image,which has been generated by the image generation unit 112, in the casedisplay region 710 of the display 101 (S820).

The input control unit 103 of the information terminal 100 detects aswipe operation performed by the user (S830). Next, the image generationunit 112 generates a thumbnail image of a similar case to be displayednext on the basis of the swipe operation. The image generation unit 112outputs the generated thumbnail image to the display control unit 104(S840).

As illustrated in FIGS. 57 and 58, the image generation unit 112generates an intermediate image, in which the thumbnail image beforeswitching of the similar case and the thumbnail image after theswitching are arranged adjacent to each other, during the swipeoperation performed with the object 350 (the user's finger) so that theintermediate image appears in accordance with the swipe operation. Theimage generation unit 112 outputs the generated intermediate image tothe display control unit 104. The display control unit 104 displays theinput intermediate image in the case display region 710 (S850).

After the swipe operation is completed, the display control unit 104displays, as illustrated in FIG. 59, the thumbnail image of the similarcase having next highest similarity across the case display region 710(S860).

In FIG. 55, the thumbnail image in the first row and the first columnillustrated in FIG. 5 is displayed across the case display region 710.If a pinch-out operation is performed, the thumbnail image is enlargedas illustrated in FIG. 56 by an enlargement ratio according to theamount of the pinch-out operation. At this time, as illustrated in FIG.56, the display control unit 104 displays the enlarged thumbnail imagesuch that the center of the CROI matches the center of the case displayregion 710.

In the second embodiment, as in the first embodiment, if a swipeoperation is performed, a position of an image is adjusted such that theCROI is located at the center of the case display region 710 asillustrated in FIGS. 58 and 59. In FIG. 59, an image obtained byenlarging the thumbnail image in the first row and the second columnillustrated in FIG. 5 is displayed in the case display region 710.

Furthermore, in the second embodiment, as can be seen from a comparisonbetween FIG. 56 and FIG. 59, an enlargement ratio of a thumbnail imageof each similar case changes such that the CROI remains the same.Details of a process performed by the image generation unit 112according to the second embodiment will be described hereinafter withreference to FIG. 60.

FIG. 60 is a block diagram illustrating a detailed configuration of theimage generation unit 112 according to the second embodiment. In FIG.60, the same components as those illustrated in FIG. 48 are given thesame reference numerals, and detailed description thereof is omitted.

The image generation unit 112 according to the second embodimentincludes the input determination section 1121, the display casedetermination section 1122, the display region determination section1123, the display image generation section 1124, a reference enlargementratio determination section 1126, an enlargement ratio determinationsection 1127, and an image enlargement section 1128.

If the input control unit 103 detects an operation performed by the userusing the operation unit 102, the input control unit 103 outputs aresult of the detection to the input determination section 1121. Theinput determination section 1121 determines the operation performed bythe user using the operation unit 102 on the basis of the results of thedetection input from the input control unit 103.

If the operation is a swipe operation, the input determination section1121 outputs information indicating that the operation is a swipeoperation and a direction of the swipe operation (for example, leftwardor rightward) to the display case determination section 1122.

If the operation is a pinch-out operation, the input determinationsection 1121 outputs information indicating that the operation is apinch-out operation and the amount of the pinch-out operation (theamount of movement of the object 350 on the display 101) to thereference enlargement ratio determination section 1126.

The reference enlargement ratio determination section 1126 determines anenlargement ratio for the thumbnail image during the pinch-out operationon the basis of the amount of the pinch-out operation (the amount ofmovement of the object 350 on the display 101) input from the inputdetermination section 1121. For example, the reference enlargement ratiodetermination section 1126 multiplies the amount of movement of theobject 350 on the display 101 by a predetermined coefficient anddetermines a product as the enlargement ratio for the thumbnail imageduring the pinch-out operation. The reference enlargement ratiodetermination section 1126 outputs the determined reference enlargementratio to the enlargement ratio determination section 1127. In thefollowing description, a thumbnail image for which a pinch-out operationhas been performed will be referred to as a “reference thumbnail image”,and an enlargement ratio determined for the thumbnail image for whichthe pinch-out operation has been performed will be referred to as a“reference enlargement ratio”.

The enlargement ratio determination section 1127 determines anenlargement ratio for a thumbnail image of a similar case to bedisplayed next on the basis of an area of a ROI of the referencethumbnail image, an area of a ROI of the similar case to be displayednext determined by the display case determination section 1122, and thereference enlargement ratio determined by the reference enlargementratio determination section 1126.

First, the enlargement ratio determination section 1127 calculates thearea of the ROI of the reference thumbnail image on the basis of the ROIinformation 4300 (FIG. 23) regarding a similar case corresponding to thereference thumbnail image. Here, if the area of the ROI of the referencethumbnail image is denoted by Sr, coordinates of an upper-left corner ofthe ROI are denoted by (xl, yt), and coordinates of a lower-right cornerof the ROI are denoted by (xr, yb), the area Sr of the ROI can becalculated using the following expression.

Sr=|xl−xr|×|yt−yb|

Next, the enlargement ratio determination section 1127 calculates thearea of the ROI from the ROI information 4300 regarding the similar caseto be displayed next. Here, if the area of the ROI of a similar case ito be displayed next is denoted by Si, coordinates of an upper-leftcorner of the ROI are denoted by (xli, yti), and coordinates of alower-right corner of the ROI are denoted by (xri, ybi), the area Si ofthe ROI can be calculated using the following expression.

Si=|xli−xri|×|yti−ybi|

Finally, the enlargement ratio determination section 1127 calculates theenlargement ratio for the similar case i to be displayed next on thebasis of the area Sr of the ROI of the reference thumbnail image, thearea Si of the ROI of the similar case i to be displayed next, and thereference enlargement ratio determined for the reference thumbnailimage. Here, if the enlargement ratio determined for the referencethumbnail image is denoted by kr, an enlargement ratio ki for thesimilar case i to be displayed next can be calculated using thefollowing expression.

ki=kr(Sr/Si)

Thus, as illustrated in FIG. 62 (referred to later), an enlargementratio with which the size of a ROI after enlargement becomes the samebetween the reference thumbnail image and the thumbnail image to bedisplayed next is calculated.

The image enlargement section 1128 enlarges the thumbnail image of thesimilar case to be displayed next determined by the display casedetermination section 1122 by the enlargement ratio determined by theenlargement ratio determination section 1127.

The display region determination section 1123 determines coordinates ofa display region of the enlarged thumbnail image on the basis of theenlargement ratio determined by the enlargement ratio determinationsection 1127 and the ROI information 4300 (FIG. 23) regarding a similarcase ID of the similar case to be displayed next such that centercoordinates of the ROI of the thumbnail image enlarged by the imageenlargement section 1128 match the center of the case display region710. The display region determined by the display region determinationsection 1123 will be described with reference to FIG. 61.

FIG. 61 is a diagram schematically illustrating a relationship betweenan enlargement ration and a display region. If the enlargement ratio iski, the image enlargement section 1128 generates an enlarged thumbnailimage illustrated in a right part of FIG. 61 from a thumbnail imageillustrated in a left part of FIG. 61.

If center coordinates of the ROI before the enlargement are denoted by(xc, yc), coordinates (ki×xc, ki×yc), which are obtained by multiplyingthe center coordinates of the ROI before the enlargement by theenlargement ratio ki, are center coordinates of the ROI after theenlargement as illustrated in FIG. 61.

As indicated in the left part of FIG. 61, a horizontal dimension and avertical dimension of a display region 720 a are denoted by w and h,respectively. In this case, a rectangular region illustrated in theright part of FIG. 61 is a display region 720 b. Coordinates of anupper-left corner of the display region 720 b are (ki×xc−w/2,ki×yc−h/2), and coordinates of a lower-right corner of the displayregion 720 b are (ki×xc+w/2, ki×yc+h/2).

FIG. 62 is a diagram schematically illustrating a reference thumbnailimage and a thumbnail image to be displayed next before and after anenlargement process. An upper-left portion of FIG. 62 illustrates thereference thumbnail image before the enlargement process. An upper-rightportion of FIG. 62 illustrates the reference thumbnail image after theenlargement process. A lower-left portion of FIG. 62 illustrates thethumbnail image to be displayed next before the enlargement process. Alower-right portion of FIG. 62 illustrates the thumbnail image to bedisplayed next after the enlargement process.

If the reference thumbnail image illustrated in the upper-left portionof FIG. 62 is enlarged by an enlargement ratio kr, the thumbnail imageillustrated in the upper-right portion of FIG. 62 is obtained. Thedisplay region determination section 1123 determines a display region720 d in the upper-right portion of FIG. 62 such that a center positionof the ROI matches a center position of the display region 720 d. Thedisplay region determination section 1123 determines the size of thedisplay region 720 d such that the size of the display region 720 dbecomes the same as the size of a display region 720 c before theenlargement process.

If the thumbnail image to be displayed next illustrated in thelower-left portion of FIG. 62 is enlarged by an enlargement ratio ki,the thumbnail image illustrated in the lower-right portion of FIG. 62 isobtained. The display region determination section 1123 determines adisplay region 720 f in the lower-right portion of FIG. 62 such that acenter position of the ROI matches a center position of the displayregion 720 f. The display region determination section 1123 determinesthe size of the display region 720 f such that the size of the displayregion 720 f becomes the same as the size of a display region 720 ebefore the enlargement process.

In FIG. 62, the enlargement ratio ki for the thumbnail image i to bedisplayed next is determined relative to the enlargement ratio kr forthe reference thumbnail image in accordance with an area ratio of theROIs. Therefore, as illustrated in FIG. 62, the sizes of the ROIs afterthe enlargement are the same.

In FIG. 60, the display image generation section 1124 generates an imageto be displayed in the case display region 710 on the basis of thecoordinates of the display region determined by the display regiondetermination section 1123 and the thumbnail image enlarged by the imageenlargement section 1128. The display image generation section 1124outputs the generated image to the display control unit 104.

As a result of the above process, according to the second embodiment, ifa thumbnail image is enlarged through, for example, a pinch-outoperation and the displayed thumbnail image is switched through a swipeoperation, a new enlarged thumbnail image displayed as a result of theswitching is displayed such that a ROI thereof is located at the centerof the case display region 710.

In general, a position of a ROI of each similar case is different.Therefore, if an enlarged thumbnail image is switched and theenlargement is canceled, an enlargement operation (pinch-out operation)needs to be performed again. On the other hand, if the enlargement isnot canceled and a new thumbnail image is displayed, a ROI might belarger than a display region or a position of the ROI might becompletely different than before. In this case, the enlargement ratioand the position of the ROI need to be changed. Therefore, if there aremany images, this operation might be physically exhausting.

According to the second embodiment, however, the doctor does not need tovisually search for a ROI each time a new thumbnail image of a similarcase is displayed. In addition, the doctor can observe a ROI immediatelywithout changing the enlargement ratio or the position of the ROI. As aresult, when thumbnail images of a plurality of similar cases aresequentially observed through the swipe operation, the observation canbe less exhausting and the reading efficiency improves, therebyimproving the efficiency of diagnosis.

The present disclosure can be used in a similar case retrieval apparatusthat provides similar cases that serve as a reference in makingdiagnoses using medical images, which are reading targets, a readingeducational apparatus for residents studying reading, and the like.

1-16. (canceled)
 17. A method comprising: causing a display to display atarget medical image being a medical image to be read; causing thedisplay to display thumbnail images in a predetermined area included inthe display, the thumbnail images corresponding one-to-one to similarmedical images, an image feature in each of the similar medical imagesand an image feature in a region of interest in the target medical imagehaving similarity, the thumbnail images including a first thumbnailimage; and when receiving selection of the first thumbnail image, (i)causing the display to remove display of the thumbnail images from thepredetermined area and (ii) causing the display to display an enlargedfirst thumbnail image in the predetermined area without the display ofthe thumbnail images in the predetermined area, the enlarged firstthumbnail image corresponding to the first thumbnail image, a size ofthe enlarged first thumbnail image being larger than a size of the firstthumbnail image.
 18. The method according to claim 17, wherein the imagefeature in each of the similar medial images is an image feature in aregion of interest in each of the similar medial images.
 19. The methodaccording to claim 17, wherein the thumbnail images correspondone-to-one to degrees of the similarity, the thumbnail images aredisplayed in descending order of the degrees of the similarity in thepredetermined area, and the enlarged first thumbnail image is generatedon the basis of a first similar medical image corresponding to the firstthumbnail image.
 20. The method according to claim 19, furthercomprising: receiving information indicating a region of interest in thetarget medical image, wherein the thumbnail images including a secondthumbnail image displayed adjacent to the first thumbnail image, andwhen detecting a swipe operation performed on the enlarged firstthumbnail image displayed in the predetermined area, (i) causing thedisplay to remove the display of the enlarged first thumbnail image fromthe predetermined area and (ii) causing the display to display a part ofan enlarged second thumbnail image in the predetermined area without thedisplay of the enlarged first thumbnail image in the predetermined area,the enlarged second thumbnail image being generated on the basis of asecond similar medical image corresponding to the second thumbnailimage, a size of the enlarged second thumbnail image being larger than asize of the second thumbnail image.
 21. A nonvolatile recording mediumstoring a program executed by an apparatus, the program causing theapparatus to perform: causing a display to display a target medicalimage being a medical image to be read; causing the display to displaythumbnail images in a predetermined area included in the display, thethumbnail images corresponding one-to-one to similar medical images, animage feature in each of the similar medical images and an image featurein a region of interest in the target medical image having similarity,the thumbnail images including a first thumbnail image; and whenreceiving selection of the first thumbnail image, (i) causing thedisplay to remove display of the thumbnail images from the predeterminedarea and (ii) causing the display to display an enlarged first thumbnailimage in the predetermined area without the display of the thumbnailimages in the predetermined area, the enlarged first thumbnail imagecorresponding to the first thumbnail image, a size of the enlarged firstthumbnail image being larger than a size of the first thumbnail image.22. The nonvolatile recording medium according to claim 21, wherein theimage feature in each of the similar medial images is an image featurein a region of interest in each of the similar medial images.
 23. Thenonvolatile recording medium according to claim 21, wherein thethumbnail images correspond one-to-one to degrees of the similarity, thethumbnail images are displayed in descending order of the degrees of thesimilarity in the predetermined area, and the enlarged first thumbnailimage is generated on the basis of a first similar medical imagecorresponding to the first thumbnail image.
 24. The nonvolatilerecording medium according to claim 23, further comprising: receivinginformation indicating a region of interest in the target medical image,wherein the thumbnail images including a second thumbnail imagedisplayed adjacent to the first thumbnail image, and when detecting aswipe operation performed on the enlarged first thumbnail imagedisplayed in the predetermined area, (i) causing the display to removethe display of the enlarged first thumbnail image from the predeterminedarea and (ii) causing the display to display a part of an enlargedsecond thumbnail image in the predetermined area without the display ofthe enlarged first thumbnail image in the predetermined area, theenlarged second thumbnail image being generated on the basis of a secondsimilar medical image corresponding to the second thumbnail image, asize of the enlarged second thumbnail image being larger than a size ofthe second thumbnail image.
 25. An apparatus comprising: a processor,wherein the processor causes a display to display a target medical imagebeing a medical image to be read, the processor causes the display todisplay thumbnail images in a predetermined area included in thedisplay, the thumbnail images corresponding one-to-one to similarmedical images, an image feature in each of the similar medical imagesand an image feature in a region of interest in the target medical imagehaving similarity, the thumbnail images including a first thumbnailimage, and when the processor receives selection of the first thumbnailimage, the processor causes the display to remove display of thethumbnail images from the predetermined area and (ii) causes the displayto display an enlarged first thumbnail image in the predetermined areawithout the display of the thumbnail images in the predetermined area,the enlarged first thumbnail image corresponding to the first thumbnailimage, a size of the enlarged first thumbnail image being larger than asize of the first thumbnail image.
 26. The apparatus according to claim25, wherein the image feature in each of the similar medial images is animage feature in a region of interest in each of the similar medialimages.
 27. The apparatus according to claim 25, wherein the thumbnailimages correspond one-to-one to degrees of the similarity, the thumbnailimages are displayed in descending order of the degrees of thesimilarity in the predetermined area, and the enlarged first thumbnailimage is generated on the basis of a first similar medical imagecorresponding to the first thumbnail image.
 28. The apparatus accordingto claim 27, wherein the processor receives information indicating aregion of interest in the target medical image, the thumbnail imagesincludes a second thumbnail image displayed adjacent to the firstthumbnail image, and when the processor detects a swipe operationperformed on the enlarged first thumbnail image displayed in thepredetermined area, (i) the processor causes the display to remove thedisplay of the enlarged first thumbnail image from the predeterminedarea and (ii) the processor causes the display to display a part of anenlarged second thumbnail image in the predetermined area without thedisplay of the enlarged first thumbnail image in the predetermined area,the enlarged second thumbnail image being generated on the basis of asecond similar medical image corresponding to the second thumbnailimage, a size of the enlarged second thumbnail image being larger than asize of the second thumbnail image.